[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:23 dist-test-slave-wjf2 krb5kdc[30738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:23 dist-test-slave-wjf2 krb5kdc[30738](info): set up 2 sockets
Sep 23 14:45:23 dist-test-slave-wjf2 krb5kdc[30738](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 14:45:24.449738 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:24 dist-test-slave-wjf2 krb5kdc[30738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:45:24.573679 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:40399
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:40399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:24.682914 30751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:24.683178 30751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:24.683220 30751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:24.687314 30751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:24.687371 30751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:24.687391 30751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:24.687407 30751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:24.687448 30751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:24.692816 30751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:40399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:40399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:24.693879 30751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:24.694975 30751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:45:24.704530 30751 server_base.cc:1034] running on GCE node
W20260923 14:45:24.704643 30759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:24.704592 30757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:24.704592 30756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:24.705497 30751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:24.706919 30751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:24.708082 30751 hybrid_clock.cc:648] HybridClock initialized: now 1790174724708060 us; error 90 us; skew 500 ppm
Sep 23 14:45:24 dist-test-slave-wjf2 krb5kdc[30738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174724, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:24.710939 30751 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:45:24.711966 30751 webserver.cc:460] Webserver started at http://127.30.2.254:34855/ using document root <none> and password file <none>
I20260923 14:45:24.712549 30751 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:24.712635 30751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:24.712872 30751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:24.714994 30751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "a10b1bd467cf4584891c7ecfbf8c23e2"
format_stamp: "Formatted at 2026-09-23 14:45:24 on dist-test-slave-wjf2"
server_key: "23ea3f5dc4886b9279aadaed52b2230e"
server_key_iv: "d6a90e8fbce2e7c1a0644a636ffbbc00"
server_key_version: "encryptionkey@0"
I20260923 14:45:24.715587 30751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "a10b1bd467cf4584891c7ecfbf8c23e2"
format_stamp: "Formatted at 2026-09-23 14:45:24 on dist-test-slave-wjf2"
server_key: "23ea3f5dc4886b9279aadaed52b2230e"
server_key_iv: "d6a90e8fbce2e7c1a0644a636ffbbc00"
server_key_version: "encryptionkey@0"
I20260923 14:45:24.720250 30751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260923 14:45:24.722982 30767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:24.724360 30751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:45:24.724502 30751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "a10b1bd467cf4584891c7ecfbf8c23e2"
format_stamp: "Formatted at 2026-09-23 14:45:24 on dist-test-slave-wjf2"
server_key: "23ea3f5dc4886b9279aadaed52b2230e"
server_key_iv: "d6a90e8fbce2e7c1a0644a636ffbbc00"
server_key_version: "encryptionkey@0"
I20260923 14:45:24.724617 30751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:24.742525 30751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:24.745631 30751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:24.745834 30751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:24.753438 30751 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:40399
I20260923 14:45:24.753451 30819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:40399 every 8 connection(s)
I20260923 14:45:24.754467 30751 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:24.757750 30820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:24.760247 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 30751
I20260923 14:45:24.760411 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:24.761664 30731 external_mini_cluster.cc:1428] Setting key 09c01577eea241b85380f0c778980924
I20260923 14:45:24.763873 30820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2: Bootstrap starting.
I20260923 14:45:24.767001 30820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:24.767879 30820 log.cc:824] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:24.769793 30820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2: No bootstrap required, opened a new log
I20260923 14:45:24.773507 30820 raft_consensus.cc:348] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10b1bd467cf4584891c7ecfbf8c23e2" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40399 } }
I20260923 14:45:24.773768 30820 raft_consensus.cc:374] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:24.773831 30820 raft_consensus.cc:729] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10b1bd467cf4584891c7ecfbf8c23e2, State: Initialized, Role: FOLLOWER
I20260923 14:45:24.774349 30820 consensus_queue.cc:260] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10b1bd467cf4584891c7ecfbf8c23e2" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40399 } }
I20260923 14:45:24.774530 30820 raft_consensus.cc:388] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:24.774626 30820 raft_consensus.cc:482] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:24.774753 30820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:24.775707 30820 raft_consensus.cc:504] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10b1bd467cf4584891c7ecfbf8c23e2" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40399 } }
I20260923 14:45:24.776098 30820 leader_election.cc:302] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a10b1bd467cf4584891c7ecfbf8c23e2; no voters:
I20260923 14:45:24.776448 30820 leader_election.cc:288] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:24.776526 30825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:24.776772 30825 raft_consensus.cc:686] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [term 1 LEADER]: Becoming Leader. State: Replica: a10b1bd467cf4584891c7ecfbf8c23e2, State: Running, Role: LEADER
I20260923 14:45:24.777112 30825 consensus_queue.cc:237] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10b1bd467cf4584891c7ecfbf8c23e2" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40399 } }
I20260923 14:45:24.777850 30820 sys_catalog.cc:564] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:24.778468 30825 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a10b1bd467cf4584891c7ecfbf8c23e2. Latest consensus state: current_term: 1 leader_uuid: "a10b1bd467cf4584891c7ecfbf8c23e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10b1bd467cf4584891c7ecfbf8c23e2" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40399 } } }
I20260923 14:45:24.778599 30825 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:24.779356 30826 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a10b1bd467cf4584891c7ecfbf8c23e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10b1bd467cf4584891c7ecfbf8c23e2" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40399 } } }
I20260923 14:45:24.779464 30826 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2 [sys.catalog]: This master's current role is: LEADER
Sep 23 14:45:24 dist-test-slave-wjf2 krb5kdc[30738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:24.779983 30830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:24.783705 30830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:24.786820 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:24.765019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.30.2.254:40399)
0923 14:45:24.765447 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:24.765462 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:24.766529 (+ 1067us) server_negotiation.cc:407] Connection header received
0923 14:45:24.769718 (+ 3189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:24.769734 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:24.770068 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:24.770390 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:24.772726 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:24.773691 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:24.775191 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:24.775654 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:24.775849 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:24.780781 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:24.780790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:24.780808 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:24.780838 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:24.782972 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:24.783600 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:24.783606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:24.783612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:24.783724 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:24.784393 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:24.784398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:24.784400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:24.784827 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:24.785042 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:24.785592 (+ 550us) server_negotiation.cc:299] Negotiation successful
0923 14:45:24.785862 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":137,"threads_started":1}
I20260923 14:45:24.790087 30830 catalog_manager.cc:1347] Generated new cluster ID: 687497f5831e4a28bd4df7f49dfd26d0
I20260923 14:45:24.790199 30830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:24.796195 30830 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:24.797430 30830 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:24.811769 30830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a10b1bd467cf4584891c7ecfbf8c23e2: Generated new TSK 0
I20260923 14:45:24.812623 30830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:45:24.850668 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40399
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:24.965008 30846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:24.965265 30846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:24.965305 30846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:24.969374 30846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:24.969429 30846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:24.969534 30846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:24.974875 30846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40399
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:24.976100 30846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:24.977239 30846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:24.987602 30854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:24.987637 30852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:24.987576 30851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:24.987737 30846 server_base.cc:1034] running on GCE node
I20260923 14:45:24.988175 30846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:24.988817 30846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:24.990005 30846 hybrid_clock.cc:648] HybridClock initialized: now 1790174724989990 us; error 30 us; skew 500 ppm
Sep 23 14:45:24 dist-test-slave-wjf2 krb5kdc[30738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174724, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:24.992796 30846 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:45:24.993665 30846 webserver.cc:460] Webserver started at http://127.30.2.193:33219/ using document root <none> and password file <none>
I20260923 14:45:24.994246 30846 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:24.994338 30846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:24.994566 30846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:24.996779 30846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "7e7f7d3d908949efa3ed7b8683cc03e6"
format_stamp: "Formatted at 2026-09-23 14:45:24 on dist-test-slave-wjf2"
server_key: "89ecdf2c311a2649323da0608283dfed"
server_key_iv: "fb1b7f8db2a26b0e768b7d78fce947d1"
server_key_version: "encryptionkey@0"
I20260923 14:45:24.997340 30846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "7e7f7d3d908949efa3ed7b8683cc03e6"
format_stamp: "Formatted at 2026-09-23 14:45:24 on dist-test-slave-wjf2"
server_key: "89ecdf2c311a2649323da0608283dfed"
server_key_iv: "fb1b7f8db2a26b0e768b7d78fce947d1"
server_key_version: "encryptionkey@0"
I20260923 14:45:25.001101 30846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:45:25.003414 30861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:25.004654 30846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:25.004793 30846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "7e7f7d3d908949efa3ed7b8683cc03e6"
format_stamp: "Formatted at 2026-09-23 14:45:24 on dist-test-slave-wjf2"
server_key: "89ecdf2c311a2649323da0608283dfed"
server_key_iv: "fb1b7f8db2a26b0e768b7d78fce947d1"
server_key_version: "encryptionkey@0"
I20260923 14:45:25.004902 30846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:25.015811 30846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:25.019100 30846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:25.019304 30846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:25.019925 30846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:25.020792 30846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:25.020864 30846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:25.020932 30846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:25.020970 30846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:25.030238 30846 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:36863
I20260923 14:45:25.030262 30975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:36863 every 8 connection(s)
I20260923 14:45:25.031219 30846 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:45:25.037158 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 30846
I20260923 14:45:25.037286 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:25.037559 30731 external_mini_cluster.cc:1428] Setting key a3c6f5061b300c6318178a4aa8a9f5c7
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[30738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174724, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:25.046550 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:25.033114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:53117 (local address 127.30.2.254:40399)
0923 14:45:25.033275 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:25.033278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:25.034123 (+ 845us) server_negotiation.cc:407] Connection header received
0923 14:45:25.035541 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:25.035544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:25.035606 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:25.035704 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:25.036944 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.037769 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.039151 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.039408 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.039494 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:25.042617 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:25.042620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:25.042622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:25.042645 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:25.044186 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:25.044903 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:25.044906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:25.044908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:25.044985 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:25.045428 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:25.045430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:25.045431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:25.045597 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:25.045669 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:25.046324 (+ 655us) server_negotiation.cc:299] Negotiation successful
0923 14:45:25.046427 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:45:25.047948 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:25.033458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40399 (local address 127.30.2.193:53117)
0923 14:45:25.033941 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:25.033994 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:25.035307 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:25.035928 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:25.035935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:25.036278 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:25.036778 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:25.036793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.037908 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.037910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:25.039009 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:25.039018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.039566 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.039572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:25.039835 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:25.040527 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:25.040554 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:25.042444 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:25.044306 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:25.044317 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:25.044338 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:25.044793 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:25.045091 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:25.045098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:25.045102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:25.045325 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:25.045704 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:25.045712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:25.046056 (+ 344us) client_negotiation.cc:769] Sending connection context
0923 14:45:25.046302 (+ 246us) client_negotiation.cc:240] Negotiation successful
0923 14:45:25.046678 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:45:25.049243 30976 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40399
I20260923 14:45:25.049464 30976 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:25.050002 30976 heartbeater.cc:511] Master 127.30.2.254:40399 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
I20260923 14:45:25.051692 30784 ts_manager.cc:194] Registered new tserver with Master: 7e7f7d3d908949efa3ed7b8683cc03e6 (127.30.2.193:36863)
I20260923 14:45:25.053061 30784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:53117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:45:25.067498 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40399
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:25.182698 30981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:25.182998 30981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:25.183094 30981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:25.187166 30981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:25.187227 30981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:25.187343 30981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:25.192678 30981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40399
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:25.193761 30981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:25.194849 30981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:45:25.205499 30981 server_base.cc:1034] running on GCE node
W20260923 14:45:25.205520 30987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:25.205544 30986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:25.205753 30989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:25.206207 30981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:25.206909 30981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:25.208124 30981 hybrid_clock.cc:648] HybridClock initialized: now 1790174725208124 us; error 47 us; skew 500 ppm
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[30738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174725, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:25.211341 30981 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:45:25.212453 30981 webserver.cc:460] Webserver started at http://127.30.2.194:33183/ using document root <none> and password file <none>
I20260923 14:45:25.213109 30981 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:25.213207 30981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:25.213451 30981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:25.215669 30981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "93e9ba717986492db936ddd9c9ce514b"
format_stamp: "Formatted at 2026-09-23 14:45:25 on dist-test-slave-wjf2"
server_key: "3e3b141bce0b1df0de5be128629cb4ff"
server_key_iv: "033d39cfebcffa5d023263baaf9d4583"
server_key_version: "encryptionkey@0"
I20260923 14:45:25.216355 30981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "93e9ba717986492db936ddd9c9ce514b"
format_stamp: "Formatted at 2026-09-23 14:45:25 on dist-test-slave-wjf2"
server_key: "3e3b141bce0b1df0de5be128629cb4ff"
server_key_iv: "033d39cfebcffa5d023263baaf9d4583"
server_key_version: "encryptionkey@0"
I20260923 14:45:25.220759 30981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:45:25.223552 30996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:25.224929 30981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:45:25.225107 30981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "93e9ba717986492db936ddd9c9ce514b"
format_stamp: "Formatted at 2026-09-23 14:45:25 on dist-test-slave-wjf2"
server_key: "3e3b141bce0b1df0de5be128629cb4ff"
server_key_iv: "033d39cfebcffa5d023263baaf9d4583"
server_key_version: "encryptionkey@0"
I20260923 14:45:25.225243 30981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:25.248031 30981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:25.251374 30981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:25.251596 30981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:25.252339 30981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:25.253230 30981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:25.253299 30981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:25.253365 30981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:25.253409 30981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:25.262518 30981 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:45499
I20260923 14:45:25.262539 31110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:45499 every 8 connection(s)
I20260923 14:45:25.263418 30981 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:25.263877 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 30981
I20260923 14:45:25.264019 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:25.264335 30731 external_mini_cluster.cc:1428] Setting key 14113e31e42137daf471cb0248b69ed5
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[30738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174725, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
I20260923 14:45:25.279959 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:25.265304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:45115 (local address 127.30.2.254:40399)
0923 14:45:25.265429 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:25.265433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:25.266481 (+ 1048us) server_negotiation.cc:407] Connection header received
0923 14:45:25.268343 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:25.268346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:25.268394 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:25.268475 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:25.270292 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.270822 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.272957 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.273199 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.273337 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:25.275831 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:25.275835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:25.275838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:25.275870 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:25.277431 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:25.278161 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:25.278165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:25.278167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:25.278277 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:25.278661 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:25.278667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:25.278672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:25.278940 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:25.279042 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:25.279611 (+ 569us) server_negotiation.cc:299] Negotiation successful
0923 14:45:25.279739 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:45:25.280941 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:25.265756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40399 (local address 127.30.2.194:45115)
0923 14:45:25.266327 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:25.266372 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:25.268069 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:25.268723 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:25.268737 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:25.269338 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:25.270112 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:25.270139 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.270959 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.270967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:25.272794 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:25.272810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.273376 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.273381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:25.273565 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:25.274216 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:25.274233 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:25.275638 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:25.277626 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:25.277631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:25.277643 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:25.277961 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:25.278403 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:25.278406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:25.278407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:25.278549 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:25.279064 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:25.279070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:25.279377 (+ 307us) client_negotiation.cc:769] Sending connection context
0923 14:45:25.279569 (+ 192us) client_negotiation.cc:240] Negotiation successful
0923 14:45:25.279944 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":150,"threads_started":1}
I20260923 14:45:25.282253 31111 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40399
I20260923 14:45:25.282543 31111 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:25.283212 31111 heartbeater.cc:511] Master 127.30.2.254:40399 requested a full tablet report, sending...
I20260923 14:45:25.284569 30784 ts_manager.cc:194] Registered new tserver with Master: 93e9ba717986492db936ddd9c9ce514b (127.30.2.194:45499)
I20260923 14:45:25.285444 30784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:45115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:45:25.294989 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40399
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:25.408478 31116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:25.408739 31116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:25.408785 31116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:25.412886 31116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:25.412943 31116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:25.413004 31116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:25.418243 31116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40399
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:25.419261 31116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:25.420356 31116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:25.430394 31122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:25.430497 31121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:25.430394 31124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:25.430711 31116 server_base.cc:1034] running on GCE node
I20260923 14:45:25.431149 31116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:25.431746 31116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:25.432942 31116 hybrid_clock.cc:648] HybridClock initialized: now 1790174725432910 us; error 46 us; skew 500 ppm
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[30738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174725, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:25.435770 31116 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:45:25.436720 31116 webserver.cc:460] Webserver started at http://127.30.2.195:45235/ using document root <none> and password file <none>
I20260923 14:45:25.437309 31116 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:25.437397 31116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:25.437628 31116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:25.439816 31116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "a35469d80978435197d77aedf8ae76dc"
format_stamp: "Formatted at 2026-09-23 14:45:25 on dist-test-slave-wjf2"
server_key: "69bcae53b0964a739ca05152858d5743"
server_key_iv: "fa70bdaf7f26f2cea052dc02ea49c738"
server_key_version: "encryptionkey@0"
I20260923 14:45:25.440407 31116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "a35469d80978435197d77aedf8ae76dc"
format_stamp: "Formatted at 2026-09-23 14:45:25 on dist-test-slave-wjf2"
server_key: "69bcae53b0964a739ca05152858d5743"
server_key_iv: "fa70bdaf7f26f2cea052dc02ea49c738"
server_key_version: "encryptionkey@0"
I20260923 14:45:25.444410 31116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 14:45:25.446830 31131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:25.448056 31116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:25.448212 31116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "a35469d80978435197d77aedf8ae76dc"
format_stamp: "Formatted at 2026-09-23 14:45:25 on dist-test-slave-wjf2"
server_key: "69bcae53b0964a739ca05152858d5743"
server_key_iv: "fa70bdaf7f26f2cea052dc02ea49c738"
server_key_version: "encryptionkey@0"
I20260923 14:45:25.448321 31116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:25.467590 31116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:25.471004 31116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:25.471218 31116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:25.471877 31116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:25.472831 31116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:25.472894 31116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:25.472952 31116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:25.472997 31116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:25.482713 31116 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:36343
I20260923 14:45:25.482753 31245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:36343 every 8 connection(s)
I20260923 14:45:25.483677 31116 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:45:25.491331 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 31116
I20260923 14:45:25.491468 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:45:25.491755 30731 external_mini_cluster.cc:1428] Setting key 439684799abc6059b68a7b78afa77d69
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[30738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174725, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:25.497848 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:25.485715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:51003 (local address 127.30.2.254:40399)
0923 14:45:25.485855 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:25.485859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:25.486733 (+ 874us) server_negotiation.cc:407] Connection header received
0923 14:45:25.488148 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:25.488151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:25.488199 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:25.488276 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:25.489619 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.490160 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.491494 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.491857 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.491991 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:25.494660 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:25.494663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:25.494666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:25.494691 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:25.496113 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:25.496591 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:25.496593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:25.496594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:25.496666 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:25.496942 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:25.496944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:25.496945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:25.497141 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:25.497218 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:25.497638 (+ 420us) server_negotiation.cc:299] Negotiation successful
0923 14:45:25.497726 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:45:25.498683 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:25.486079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40399 (local address 127.30.2.195:51003)
0923 14:45:25.486554 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:25.486600 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:25.487932 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:25.488546 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:25.488556 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:25.488925 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:25.489452 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:25.489468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.490288 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.490291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:25.491335 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:25.491349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.492064 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.492112 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:25.492305 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:25.492938 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:25.492955 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:25.494494 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:25.496232 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:25.496237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:25.496249 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:25.496500 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:25.496738 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:25.496740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:25.496742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:25.496868 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:25.497225 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:25.497230 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:25.497445 (+ 215us) client_negotiation.cc:769] Sending connection context
0923 14:45:25.497595 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 14:45:25.497845 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":125,"threads_started":1}
I20260923 14:45:25.499740 31246 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40399
I20260923 14:45:25.499979 31246 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:25.500411 31246 heartbeater.cc:511] Master 127.30.2.254:40399 requested a full tablet report, sending...
I20260923 14:45:25.501432 30784 ts_manager.cc:194] Registered new tserver with Master: a35469d80978435197d77aedf8ae76dc (127.30.2.195:36343)
I20260923 14:45:25.502002 30784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:51003
I20260923 14:45:25.506197 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[30738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[30738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:45:25.532943 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:25.522831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.30.2.193:36863)
0923 14:45:25.523124 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:25.523141 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:25.523177 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:45:25.523287 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:25.523292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:25.523493 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:25.523612 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:25.524403 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.525193 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.526066 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.526527 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.526646 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:25.529350 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:25.529359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:25.529374 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:25.529398 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:25.531096 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:25.531482 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:25.531487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:25.531488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:25.531566 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:25.531857 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:25.531860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:25.531861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:25.532150 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:25.532369 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:25.532616 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 14:45:25.532775 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":117,"threads_started":1}
W20260923 14:45:25.533957 30889 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56026
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[30738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[30738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:45:25.559916 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:25.551255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.30.2.193:36863)
0923 14:45:25.551417 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:25.551421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:25.551482 (+ 61us) server_negotiation.cc:407] Connection header received
0923 14:45:25.551663 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:25.551666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:25.551707 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:25.551854 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:25.552645 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.553226 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.554117 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:25.554346 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:25.554430 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:25.556903 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:25.556906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:25.556908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:25.556930 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:25.558198 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:25.558555 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:25.558557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:25.558558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:25.558630 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:25.558845 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:25.558847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:25.558848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:25.558993 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:25.559112 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:25.559341 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 14:45:25.559631 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 14:45:25.561271 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 30846
I20260923 14:45:25.568006 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 30981
I20260923 14:45:25.574666 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 31116
I20260923 14:45:25.580873 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 30751
2026-09-23T14:45:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2234 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[31260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[31260](info): set up 2 sockets
Sep 23 14:45:25 dist-test-slave-wjf2 krb5kdc[31260](info): commencing operation
krb5kdc: starting...
W20260923 14:45:26.716714 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:26 dist-test-slave-wjf2 krb5kdc[31260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:45:26.842191 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:34527
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:34527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:26.955202 31273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:26.955440 31273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:26.955476 31273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:26.959434 31273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:26.959481 31273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:26.959501 31273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:26.959517 31273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:26.959560 31273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:26.964726 31273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:34527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:34527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:26.965668 31273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:26.966656 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:26.975054 31281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:26.975054 31279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:26.975070 31278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:26.975337 31273 server_base.cc:1034] running on GCE node
I20260923 14:45:26.975940 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:26.977087 31273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:26.978247 31273 hybrid_clock.cc:648] HybridClock initialized: now 1790174726978227 us; error 32 us; skew 500 ppm
Sep 23 14:45:26 dist-test-slave-wjf2 krb5kdc[31260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174726, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:26.980989 31273 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:45:26.981854 31273 webserver.cc:460] Webserver started at http://127.30.2.254:36641/ using document root <none> and password file <none>
I20260923 14:45:26.982401 31273 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:26.982461 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:26.982689 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:26.984864 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "d404d74232f64b948d525070000b8712"
format_stamp: "Formatted at 2026-09-23 14:45:26 on dist-test-slave-wjf2"
server_key: "eaf492b54c6a453df061f41683a696cd"
server_key_iv: "2bd84f9a7b6a26d4f1175478c3881105"
server_key_version: "encryptionkey@0"
I20260923 14:45:26.985410 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "d404d74232f64b948d525070000b8712"
format_stamp: "Formatted at 2026-09-23 14:45:26 on dist-test-slave-wjf2"
server_key: "eaf492b54c6a453df061f41683a696cd"
server_key_iv: "2bd84f9a7b6a26d4f1175478c3881105"
server_key_version: "encryptionkey@0"
I20260923 14:45:26.989243 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:45:26.991520 31288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:26.992769 31273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:26.992916 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "d404d74232f64b948d525070000b8712"
format_stamp: "Formatted at 2026-09-23 14:45:26 on dist-test-slave-wjf2"
server_key: "eaf492b54c6a453df061f41683a696cd"
server_key_iv: "2bd84f9a7b6a26d4f1175478c3881105"
server_key_version: "encryptionkey@0"
I20260923 14:45:26.993026 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:27.013810 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:27.017325 31273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:27.017553 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:27.025388 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:34527
I20260923 14:45:27.025416 31341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:34527 every 8 connection(s)
I20260923 14:45:27.026388 31273 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:27.028134 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 31273
I20260923 14:45:27.028240 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:27.028506 30731 external_mini_cluster.cc:1428] Setting key c0deb89f66406f17da4bde3ca98cbce7
I20260923 14:45:27.029594 31342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:27.035306 31342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712: Bootstrap starting.
Sep 23 14:45:27 dist-test-slave-wjf2 krb5kdc[31260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:27.037832 31342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:27.038586 31342 log.cc:824] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:27.040556 31342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712: No bootstrap required, opened a new log
I20260923 14:45:27.042520 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.029959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.30.2.254:34527)
0923 14:45:27.030423 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:27.030438 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:27.030476 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:45:27.031111 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:27.031128 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:27.031444 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:27.031838 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:27.032718 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.033740 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.034610 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.035068 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.035242 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.037994 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:27.038000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:27.038013 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:27.038043 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:27.039835 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.040282 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.040288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.040294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.040405 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.040683 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.040687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.040689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.041073 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:27.041280 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:27.041550 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 14:45:27.041827 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":146,"threads_started":1}
I20260923 14:45:27.044160 31342 raft_consensus.cc:348] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d404d74232f64b948d525070000b8712" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 34527 } }
I20260923 14:45:27.044389 31342 raft_consensus.cc:374] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:27.044453 31342 raft_consensus.cc:729] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d404d74232f64b948d525070000b8712, State: Initialized, Role: FOLLOWER
I20260923 14:45:27.044947 31342 consensus_queue.cc:260] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d404d74232f64b948d525070000b8712" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 34527 } }
I20260923 14:45:27.045099 31342 raft_consensus.cc:388] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:27.045181 31342 raft_consensus.cc:482] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:27.045276 31342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:27.046226 31342 raft_consensus.cc:504] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d404d74232f64b948d525070000b8712" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 34527 } }
I20260923 14:45:27.046581 31342 leader_election.cc:302] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d404d74232f64b948d525070000b8712; no voters:
I20260923 14:45:27.046918 31342 leader_election.cc:288] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:27.047079 31347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:27.047348 31347 raft_consensus.cc:686] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [term 1 LEADER]: Becoming Leader. State: Replica: d404d74232f64b948d525070000b8712, State: Running, Role: LEADER
I20260923 14:45:27.047653 31347 consensus_queue.cc:237] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d404d74232f64b948d525070000b8712" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 34527 } }
I20260923 14:45:27.048158 31342 sys_catalog.cc:564] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:27.049291 31348 sys_catalog.cc:455] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d404d74232f64b948d525070000b8712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d404d74232f64b948d525070000b8712" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 34527 } } }
I20260923 14:45:27.049433 31348 sys_catalog.cc:458] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:27.049741 31349 sys_catalog.cc:455] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d404d74232f64b948d525070000b8712. Latest consensus state: current_term: 1 leader_uuid: "d404d74232f64b948d525070000b8712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d404d74232f64b948d525070000b8712" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 34527 } } }
I20260923 14:45:27.049860 31349 sys_catalog.cc:458] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:27.049795 31356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:27.052872 31356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:27.057682 31356 catalog_manager.cc:1347] Generated new cluster ID: fc5922b87d22426a85e2bd4d9f46745c
I20260923 14:45:27.057744 31356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:27.073133 31356 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:27.074009 31356 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:27.079522 31356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d404d74232f64b948d525070000b8712: Generated new TSK 0
I20260923 14:45:27.080241 31356 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:45:27.110795 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:34527
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:27.232249 31368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:27.232641 31368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:27.232726 31368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:27.237345 31368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:27.237434 31368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:27.237525 31368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:27.246325 31368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:34527
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:27.248049 31368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:27.249410 31368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:27.259323 31373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:27.259352 31376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:27.259384 31374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:27.259555 31368 server_base.cc:1034] running on GCE node
I20260923 14:45:27.260129 31368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:27.260789 31368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:27.261972 31368 hybrid_clock.cc:648] HybridClock initialized: now 1790174727261936 us; error 46 us; skew 500 ppm
Sep 23 14:45:27 dist-test-slave-wjf2 krb5kdc[31260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174727, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:27.265039 31368 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:45:27.266039 31368 webserver.cc:460] Webserver started at http://127.30.2.193:34835/ using document root <none> and password file <none>
I20260923 14:45:27.266645 31368 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:27.266707 31368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:27.266938 31368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:27.269244 31368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f"
format_stamp: "Formatted at 2026-09-23 14:45:27 on dist-test-slave-wjf2"
server_key: "87dc727ccdbc766347df00cced3a62fd"
server_key_iv: "0696c0331537fca8d05221c5a7262c54"
server_key_version: "encryptionkey@0"
I20260923 14:45:27.269874 31368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f"
format_stamp: "Formatted at 2026-09-23 14:45:27 on dist-test-slave-wjf2"
server_key: "87dc727ccdbc766347df00cced3a62fd"
server_key_iv: "0696c0331537fca8d05221c5a7262c54"
server_key_version: "encryptionkey@0"
I20260923 14:45:27.274827 31368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 14:45:27.278081 31383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:27.279459 31368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:45:27.279584 31368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f"
format_stamp: "Formatted at 2026-09-23 14:45:27 on dist-test-slave-wjf2"
server_key: "87dc727ccdbc766347df00cced3a62fd"
server_key_iv: "0696c0331537fca8d05221c5a7262c54"
server_key_version: "encryptionkey@0"
I20260923 14:45:27.279697 31368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:27.327301 31368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:27.332273 31368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:27.332726 31368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:27.333550 31368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:27.334794 31368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:27.334873 31368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:27.334930 31368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:27.334961 31368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:27.345367 31368 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:37679
I20260923 14:45:27.345389 31498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:37679 every 8 connection(s)
I20260923 14:45:27.346282 31368 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:45:27.347895 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 31368
I20260923 14:45:27.348007 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:27.348313 30731 external_mini_cluster.cc:1428] Setting key adf65856e7965c496df52ae6c71048d7
Sep 23 14:45:27 dist-test-slave-wjf2 krb5kdc[31260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174727, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:45:27.361395 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.348177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:34473 (local address 127.30.2.254:34527)
0923 14:45:27.348351 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:27.348356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:27.349418 (+ 1062us) server_negotiation.cc:407] Connection header received
0923 14:45:27.350601 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:27.350604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:27.350644 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:27.350705 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:27.352321 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.352854 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.354638 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.354883 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.354968 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.357700 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:27.357706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:27.357709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:27.357739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:27.359394 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.359946 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.359949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.359950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.360023 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.360352 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.360355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.360356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.360527 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:27.360596 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:27.361075 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 14:45:27.361192 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
I20260923 14:45:27.362236 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.348733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:34527 (local address 127.30.2.193:34473)
0923 14:45:27.349284 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:27.349327 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:27.350338 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:27.350893 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:27.350922 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:27.351473 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:27.352177 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.352192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.352985 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.352990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.354479 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.354494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.355044 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.355048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.355260 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.355901 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:27.355924 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:27.357537 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:27.359507 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:27.359512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:27.359523 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:27.359771 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:27.360129 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:27.360132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:27.360133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:27.360264 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:27.360610 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:27.360615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:27.360871 (+ 256us) client_negotiation.cc:769] Sending connection context
0923 14:45:27.361016 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 14:45:27.361339 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":115,"threads_started":1}
I20260923 14:45:27.363349 31499 heartbeater.cc:347] Connected to a master server at 127.30.2.254:34527
I20260923 14:45:27.363590 31499 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:27.364121 31499 heartbeater.cc:511] Master 127.30.2.254:34527 requested a full tablet report, sending...
I20260923 14:45:27.365808 31305 ts_manager.cc:194] Registered new tserver with Master: 2dadb6be886d4e33b8ed2b5057ab2f5f (127.30.2.193:37679)
I20260923 14:45:27.367511 31305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:34473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:45:27.380506 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:34527
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:27.497058 31504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:27.497346 31504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:27.497393 31504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:27.501338 31504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:27.501400 31504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:27.501459 31504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:27.506839 31504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:34527
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:27.508038 31504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:27.509217 31504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:27.520555 31512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:27.520555 31510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:27.520555 31509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:27.520830 31504 server_base.cc:1034] running on GCE node
I20260923 14:45:27.521245 31504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:27.521900 31504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:27.523073 31504 hybrid_clock.cc:648] HybridClock initialized: now 1790174727523047 us; error 37 us; skew 500 ppm
Sep 23 14:45:27 dist-test-slave-wjf2 krb5kdc[31260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174727, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:27.525882 31504 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:45:27.526716 31504 webserver.cc:460] Webserver started at http://127.30.2.194:41303/ using document root <none> and password file <none>
I20260923 14:45:27.527295 31504 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:27.527354 31504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:27.527595 31504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:27.529726 31504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "83bf3b587b564a1d88581a50d7f2d66b"
format_stamp: "Formatted at 2026-09-23 14:45:27 on dist-test-slave-wjf2"
server_key: "d681ebb0ea869b1fa7ecc33576382982"
server_key_iv: "14d0374582ffff4f07f5f91082f59c35"
server_key_version: "encryptionkey@0"
I20260923 14:45:27.530319 31504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "83bf3b587b564a1d88581a50d7f2d66b"
format_stamp: "Formatted at 2026-09-23 14:45:27 on dist-test-slave-wjf2"
server_key: "d681ebb0ea869b1fa7ecc33576382982"
server_key_iv: "14d0374582ffff4f07f5f91082f59c35"
server_key_version: "encryptionkey@0"
I20260923 14:45:27.534087 31504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:45:27.536491 31519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:27.537685 31504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:27.537797 31504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "83bf3b587b564a1d88581a50d7f2d66b"
format_stamp: "Formatted at 2026-09-23 14:45:27 on dist-test-slave-wjf2"
server_key: "d681ebb0ea869b1fa7ecc33576382982"
server_key_iv: "14d0374582ffff4f07f5f91082f59c35"
server_key_version: "encryptionkey@0"
I20260923 14:45:27.537890 31504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:27.552675 31504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:27.555831 31504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:27.556006 31504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:27.556572 31504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:27.557338 31504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:27.557381 31504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:27.557440 31504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:27.557473 31504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:27.566398 31504 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:41027
I20260923 14:45:27.566406 31633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:41027 every 8 connection(s)
I20260923 14:45:27.567267 31504 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
Sep 23 14:45:27 dist-test-slave-wjf2 krb5kdc[31260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174727, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:27.576681 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 31504
I20260923 14:45:27.576900 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:27.577158 30731 external_mini_cluster.cc:1428] Setting key fcabc19ac0acb1358dc6e91f5c1203a8
I20260923 14:45:27.580439 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.569110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:36663 (local address 127.30.2.254:34527)
0923 14:45:27.569242 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:27.569246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:27.570071 (+ 825us) server_negotiation.cc:407] Connection header received
0923 14:45:27.570894 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:27.570896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:27.570942 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:27.571016 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:27.572292 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.572839 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.573995 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.574227 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.574314 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.576700 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:27.576704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:27.576706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:27.576737 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:27.578362 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.578872 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.578876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.578878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.578956 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.579353 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.579357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.579359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.579539 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:27.579644 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:27.580168 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 14:45:27.580285 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:45:27.581235 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.569474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:34527 (local address 127.30.2.194:36663)
0923 14:45:27.569921 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:27.569961 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:27.570704 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:27.571210 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:27.571217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:27.571593 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:27.572123 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.572137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.572977 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.572979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.573867 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.573876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.574366 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.574369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.574504 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.575049 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:27.575065 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:27.576529 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:27.578495 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:27.578501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:27.578515 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:27.578766 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:27.579064 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:27.579067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:27.579069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:27.579226 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:27.579643 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:27.579649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:27.579942 (+ 293us) client_negotiation.cc:769] Sending connection context
0923 14:45:27.580126 (+ 184us) client_negotiation.cc:240] Negotiation successful
0923 14:45:27.580437 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":130,"threads_started":1}
I20260923 14:45:27.582480 31634 heartbeater.cc:347] Connected to a master server at 127.30.2.254:34527
I20260923 14:45:27.582736 31634 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:27.583295 31634 heartbeater.cc:511] Master 127.30.2.254:34527 requested a full tablet report, sending...
I20260923 14:45:27.584776 31305 ts_manager.cc:194] Registered new tserver with Master: 83bf3b587b564a1d88581a50d7f2d66b (127.30.2.194:41027)
I20260923 14:45:27.585655 31305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:36663
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:45:27.606907 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:34527
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:27.714931 31639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:27.715175 31639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:27.715219 31639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:27.719139 31639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:27.719192 31639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:27.719276 31639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:27.724387 31639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:34527
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:27.725361 31639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:27.726334 31639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:27.735886 31645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:27.735989 31639 server_base.cc:1034] running on GCE node
W20260923 14:45:27.735893 31644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:27.735900 31647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:27.736532 31639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:27.737120 31639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:27.738333 31639 hybrid_clock.cc:648] HybridClock initialized: now 1790174727738315 us; error 27 us; skew 500 ppm
Sep 23 14:45:27 dist-test-slave-wjf2 krb5kdc[31260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174727, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:27.741183 31639 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:45:27.742187 31639 webserver.cc:460] Webserver started at http://127.30.2.195:33017/ using document root <none> and password file <none>
I20260923 14:45:27.742753 31639 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:27.742843 31639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:27.743074 31639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:27.745177 31639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "1e191f03734e4704b93f1262a9fe974a"
format_stamp: "Formatted at 2026-09-23 14:45:27 on dist-test-slave-wjf2"
server_key: "c0cbc92877d20988c7d1fa7db9776fe5"
server_key_iv: "76fd600f4483527d41eb53f2c7fae2ff"
server_key_version: "encryptionkey@0"
I20260923 14:45:27.745752 31639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "1e191f03734e4704b93f1262a9fe974a"
format_stamp: "Formatted at 2026-09-23 14:45:27 on dist-test-slave-wjf2"
server_key: "c0cbc92877d20988c7d1fa7db9776fe5"
server_key_iv: "76fd600f4483527d41eb53f2c7fae2ff"
server_key_version: "encryptionkey@0"
I20260923 14:45:27.749650 31639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:45:27.752190 31654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:27.753366 31639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:27.753507 31639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "1e191f03734e4704b93f1262a9fe974a"
format_stamp: "Formatted at 2026-09-23 14:45:27 on dist-test-slave-wjf2"
server_key: "c0cbc92877d20988c7d1fa7db9776fe5"
server_key_iv: "76fd600f4483527d41eb53f2c7fae2ff"
server_key_version: "encryptionkey@0"
I20260923 14:45:27.753619 31639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:27.779670 31639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:27.783205 31639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:27.783416 31639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:27.784128 31639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:27.785022 31639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:27.785107 31639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:27.785175 31639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:27.785204 31639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:27.793893 31639 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:40975
I20260923 14:45:27.793923 31767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:40975 every 8 connection(s)
I20260923 14:45:27.794767 31639 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:45:27.802627 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 31639
I20260923 14:45:27.802762 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:45:27.803023 30731 external_mini_cluster.cc:1428] Setting key eae1e3025df823a2edfbd057935d45cf
Sep 23 14:45:27 dist-test-slave-wjf2 krb5kdc[31260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174727, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:27.809372 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.796694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:59031 (local address 127.30.2.254:34527)
0923 14:45:27.796832 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:27.796837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:27.797738 (+ 901us) server_negotiation.cc:407] Connection header received
0923 14:45:27.798743 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:27.798747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:27.798811 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:27.798918 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:27.800212 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.800795 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.802025 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.802285 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.802405 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.805558 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:27.805562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:27.805566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:27.805599 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:27.807021 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.807625 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.807631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.807633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.807727 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.808143 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.808149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.808151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.808359 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:27.808485 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:27.809032 (+ 547us) server_negotiation.cc:299] Negotiation successful
0923 14:45:27.809179 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:45:27.810163 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.797041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:34527 (local address 127.30.2.195:59031)
0923 14:45:27.797554 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:27.797599 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:27.798503 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:27.799104 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:27.799112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:27.799480 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:27.800058 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.800073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.800928 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.800938 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.801889 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.801899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.802448 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.802452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.802640 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.803494 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:27.803514 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:27.805369 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:27.807184 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:27.807192 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:27.807208 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:27.807497 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:27.807866 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:27.807870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:27.807873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:27.808029 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:27.808483 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:27.808491 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:27.808757 (+ 266us) client_negotiation.cc:769] Sending connection context
0923 14:45:27.808982 (+ 225us) client_negotiation.cc:240] Negotiation successful
0923 14:45:27.809318 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":159,"threads_started":1}
I20260923 14:45:27.811352 31768 heartbeater.cc:347] Connected to a master server at 127.30.2.254:34527
I20260923 14:45:27.811568 31768 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:27.812068 31768 heartbeater.cc:511] Master 127.30.2.254:34527 requested a full tablet report, sending...
I20260923 14:45:27.813050 31305 ts_manager.cc:194] Registered new tserver with Master: 1e191f03734e4704b93f1262a9fe974a (127.30.2.195:40975)
I20260923 14:45:27.813620 31305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:59031
I20260923 14:45:27.816994 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:45:27.828243 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.820574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.30.2.254:34527)
0923 14:45:27.820690 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:27.820694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:27.820912 (+ 218us) server_negotiation.cc:407] Connection header received
0923 14:45:27.821009 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:27.821011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:27.821049 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:27.821147 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:27.821856 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.822387 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.823357 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.823565 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.823644 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.825264 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:27.825267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:27.825269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:27.825288 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:27.826555 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.826964 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.826967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.826968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.827037 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.827279 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.827281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.827282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.827476 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:27.827579 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:27.828026 (+ 447us) server_negotiation.cc:299] Negotiation successful
0923 14:45:27.828120 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:45:27.833276 31305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:45:27.836297 31305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:27.851336 31785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.846281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.30.2.195:40975)
0923 14:45:27.846573 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:27.846595 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:27.847001 (+ 406us) server_negotiation.cc:407] Connection header received
0923 14:45:27.847086 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:27.847091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:27.847278 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:27.847424 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:27.847622 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.848635 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.850053 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.850638 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.850756 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.850905 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:27.851080 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 14:45:27.851183 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":127,"threads_started":1}
I20260923 14:45:27.851332 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.845705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:40975 (local address 127.0.0.1:56254)
0923 14:45:27.846869 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:27.846879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:27.846977 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:27.847405 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:27.847409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:27.847428 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:27.847524 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.847529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.848810 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.848816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.849913 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.849924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.850768 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.850771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.850880 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.850918 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 14:45:27.851046 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 14:45:27.851167 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":157,"thread_start_us":112,"threads_started":1}
I20260923 14:45:27.851346 31781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.844992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:37679 (local address 127.0.0.1:57034)
0923 14:45:27.845418 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:27.845462 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:27.845593 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:27.846523 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:27.846530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:27.846551 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:27.846698 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.846707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.848342 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.848345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.849825 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.849837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.850934 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.850937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.851020 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.851042 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:45:27.851114 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 14:45:27.851182 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":83,"thread_start_us":82,"threads_started":1}
I20260923 14:45:27.851642 31783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.845570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.30.2.194:41027)
0923 14:45:27.845940 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:27.845952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:27.846191 (+ 239us) server_negotiation.cc:407] Connection header received
0923 14:45:27.846267 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:27.846272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:27.846420 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:27.846555 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:27.846983 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.848086 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.849982 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.850795 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.850911 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.851048 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:27.851408 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 14:45:27.851504 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":150,"threads_started":1}
I20260923 14:45:27.851905 31784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.845043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41027 (local address 127.0.0.1:47312)
0923 14:45:27.845984 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:27.845999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:27.846097 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:27.846725 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:27.846729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:27.846751 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:27.846858 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.846864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.848206 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.848213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.849825 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.849870 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.851120 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.851123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.851201 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.851220 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 14:45:27.851284 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 14:45:27.851338 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":117,"thread_start_us":99,"threads_started":1}
I20260923 14:45:27.852003 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.845130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.30.2.193:37679)
0923 14:45:27.845946 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:27.845961 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:27.845984 (+ 23us) server_negotiation.cc:407] Connection header received
0923 14:45:27.846070 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:27.846076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:27.846337 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:27.846862 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:27.847030 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.848220 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.849975 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.850795 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.851533 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.851684 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:27.851735 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 14:45:27.851851 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":105,"threads_started":1}
I20260923 14:45:27.854435 31567 tablet_service.cc:1467] Processing CreateTablet for tablet dbbd97a0da5d4207ab0cfe57393f5179 (DEFAULT_TABLE table=test-table [id=8698fa0b8fb2499a9310a4007b9864e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:27.854619 31432 tablet_service.cc:1467] Processing CreateTablet for tablet dbbd97a0da5d4207ab0cfe57393f5179 (DEFAULT_TABLE table=test-table [id=8698fa0b8fb2499a9310a4007b9864e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:27.854624 31702 tablet_service.cc:1467] Processing CreateTablet for tablet dbbd97a0da5d4207ab0cfe57393f5179 (DEFAULT_TABLE table=test-table [id=8698fa0b8fb2499a9310a4007b9864e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:27.855460 31567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbd97a0da5d4207ab0cfe57393f5179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:27.855640 31702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbd97a0da5d4207ab0cfe57393f5179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:27.855640 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbd97a0da5d4207ab0cfe57393f5179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:27.861581 31786 tablet_bootstrap.cc:492] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b: Bootstrap starting.
I20260923 14:45:27.862952 31787 tablet_bootstrap.cc:492] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f: Bootstrap starting.
I20260923 14:45:27.864094 31788 tablet_bootstrap.cc:492] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a: Bootstrap starting.
I20260923 14:45:27.864511 31786 tablet_bootstrap.cc:654] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:27.865001 31787 tablet_bootstrap.cc:654] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:27.865272 31786 log.cc:824] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:27.865805 31787 log.cc:824] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:27.866091 31788 tablet_bootstrap.cc:654] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:27.867143 31786 tablet_bootstrap.cc:492] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b: No bootstrap required, opened a new log
I20260923 14:45:27.867163 31788 log.cc:824] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:27.867342 31786 ts_tablet_manager.cc:1397] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:45:27.867800 31787 tablet_bootstrap.cc:492] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f: No bootstrap required, opened a new log
I20260923 14:45:27.867974 31787 ts_tablet_manager.cc:1397] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:45:27.869052 31788 tablet_bootstrap.cc:492] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a: No bootstrap required, opened a new log
I20260923 14:45:27.869246 31788 ts_tablet_manager.cc:1397] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:45:27.872018 31786 raft_consensus.cc:348] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37679 } } peers { permanent_uuid: "83bf3b587b564a1d88581a50d7f2d66b" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41027 } } peers { permanent_uuid: "1e191f03734e4704b93f1262a9fe974a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40975 } }
I20260923 14:45:27.871991 31788 raft_consensus.cc:348] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37679 } } peers { permanent_uuid: "83bf3b587b564a1d88581a50d7f2d66b" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41027 } } peers { permanent_uuid: "1e191f03734e4704b93f1262a9fe974a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40975 } }
I20260923 14:45:27.872291 31786 raft_consensus.cc:374] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:27.872290 31788 raft_consensus.cc:374] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:27.872340 31786 raft_consensus.cc:729] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83bf3b587b564a1d88581a50d7f2d66b, State: Initialized, Role: FOLLOWER
I20260923 14:45:27.872351 31788 raft_consensus.cc:729] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e191f03734e4704b93f1262a9fe974a, State: Initialized, Role: FOLLOWER
I20260923 14:45:27.872802 31786 consensus_queue.cc:260] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37679 } } peers { permanent_uuid: "83bf3b587b564a1d88581a50d7f2d66b" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41027 } } peers { permanent_uuid: "1e191f03734e4704b93f1262a9fe974a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40975 } }
I20260923 14:45:27.872802 31788 consensus_queue.cc:260] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37679 } } peers { permanent_uuid: "83bf3b587b564a1d88581a50d7f2d66b" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41027 } } peers { permanent_uuid: "1e191f03734e4704b93f1262a9fe974a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40975 } }
I20260923 14:45:27.872956 31787 raft_consensus.cc:348] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37679 } } peers { permanent_uuid: "83bf3b587b564a1d88581a50d7f2d66b" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41027 } } peers { permanent_uuid: "1e191f03734e4704b93f1262a9fe974a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40975 } }
I20260923 14:45:27.873198 31787 raft_consensus.cc:374] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:27.873257 31787 raft_consensus.cc:729] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dadb6be886d4e33b8ed2b5057ab2f5f, State: Initialized, Role: FOLLOWER
I20260923 14:45:27.873598 31768 heartbeater.cc:503] Master 127.30.2.254:34527 was elected leader, sending a full tablet report...
I20260923 14:45:27.873626 31787 consensus_queue.cc:260] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37679 } } peers { permanent_uuid: "83bf3b587b564a1d88581a50d7f2d66b" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41027 } } peers { permanent_uuid: "1e191f03734e4704b93f1262a9fe974a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40975 } }
I20260923 14:45:27.873888 31786 ts_tablet_manager.cc:1428] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260923 14:45:27.873919 31634 heartbeater.cc:503] Master 127.30.2.254:34527 was elected leader, sending a full tablet report...
I20260923 14:45:27.874385 31787 ts_tablet_manager.cc:1428] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 14:45:27.874675 31499 heartbeater.cc:503] Master 127.30.2.254:34527 was elected leader, sending a full tablet report...
I20260923 14:45:27.875157 31788 ts_tablet_manager.cc:1428] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:45:27.947969 31793 raft_consensus.cc:482] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:27.948215 31793 raft_consensus.cc:504] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37679 } } peers { permanent_uuid: "83bf3b587b564a1d88581a50d7f2d66b" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41027 } } peers { permanent_uuid: "1e191f03734e4704b93f1262a9fe974a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40975 } }
I20260923 14:45:27.949476 31793 leader_election.cc:288] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dadb6be886d4e33b8ed2b5057ab2f5f (127.30.2.193:37679), 1e191f03734e4704b93f1262a9fe974a (127.30.2.195:40975)
I20260923 14:45:27.953457 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.949730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:40975 (local address 127.30.2.194:47239)
0923 14:45:27.949872 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:27.949890 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:27.949999 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:27.950342 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:27.950345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:27.950363 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:27.950502 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.950508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.951228 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.951231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.952136 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.952146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.953013 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.953017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.953125 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.953153 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 14:45:27.953241 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 14:45:27.953310 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"spinlock_wait_cycles":10624}
I20260923 14:45:27.953469 31785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.949872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:47239 (local address 127.30.2.195:40975)
0923 14:45:27.950009 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:27.950013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:27.950031 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:45:27.950090 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:27.950162 (+ 72us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:27.950232 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:27.950326 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:27.950609 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.951115 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.952267 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.952877 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.953018 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.953077 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:27.953289 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 14:45:27.953359 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:45:27.954166 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.949865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:55203 (local address 127.30.2.193:37679)
0923 14:45:27.950045 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:27.950051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:27.950371 (+ 320us) server_negotiation.cc:407] Connection header received
0923 14:45:27.950547 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:27.950550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:27.950595 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:27.950673 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:27.950920 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.951414 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.953011 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.953638 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.953771 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.953817 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:27.953984 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 14:45:27.954055 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 14:45:27.954177 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.949730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:37679 (local address 127.30.2.194:55203)
0923 14:45:27.950282 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:27.950294 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:27.950435 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:27.950684 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:27.950687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:27.950701 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:27.950819 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.950825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.951525 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.951529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.952893 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:27.952905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.953760 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.953763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:27.953863 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.953886 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 14:45:27.953968 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 14:45:27.954035 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":134,"spinlock_wait_cycles":5760,"thread_start_us":135,"threads_started":1}
I20260923 14:45:27.954348 31722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbd97a0da5d4207ab0cfe57393f5179" candidate_uuid: "83bf3b587b564a1d88581a50d7f2d66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e191f03734e4704b93f1262a9fe974a" is_pre_election: true
I20260923 14:45:27.954753 31722 raft_consensus.cc:2454] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83bf3b587b564a1d88581a50d7f2d66b in term 0.
I20260923 14:45:27.955096 31452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbd97a0da5d4207ab0cfe57393f5179" candidate_uuid: "83bf3b587b564a1d88581a50d7f2d66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f" is_pre_election: true
I20260923 14:45:27.955327 31520 leader_election.cc:302] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e191f03734e4704b93f1262a9fe974a, 83bf3b587b564a1d88581a50d7f2d66b; no voters:
I20260923 14:45:27.955375 31452 raft_consensus.cc:2454] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83bf3b587b564a1d88581a50d7f2d66b in term 0.
I20260923 14:45:27.955583 31793 raft_consensus.cc:2781] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:27.955698 31793 raft_consensus.cc:482] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:27.955765 31793 raft_consensus.cc:3037] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:27.956828 31793 raft_consensus.cc:504] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37679 } } peers { permanent_uuid: "83bf3b587b564a1d88581a50d7f2d66b" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41027 } } peers { permanent_uuid: "1e191f03734e4704b93f1262a9fe974a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40975 } }
I20260923 14:45:27.957186 31793 leader_election.cc:288] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [CANDIDATE]: Term 1 election: Requested vote from peers 2dadb6be886d4e33b8ed2b5057ab2f5f (127.30.2.193:37679), 1e191f03734e4704b93f1262a9fe974a (127.30.2.195:40975)
I20260923 14:45:27.957593 31722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbd97a0da5d4207ab0cfe57393f5179" candidate_uuid: "83bf3b587b564a1d88581a50d7f2d66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e191f03734e4704b93f1262a9fe974a"
I20260923 14:45:27.957612 31452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbd97a0da5d4207ab0cfe57393f5179" candidate_uuid: "83bf3b587b564a1d88581a50d7f2d66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f"
I20260923 14:45:27.957742 31722 raft_consensus.cc:3037] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:27.957742 31452 raft_consensus.cc:3037] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:27.958818 31452 raft_consensus.cc:2454] T dbbd97a0da5d4207ab0cfe57393f5179 P 2dadb6be886d4e33b8ed2b5057ab2f5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83bf3b587b564a1d88581a50d7f2d66b in term 1.
I20260923 14:45:27.959085 31722 raft_consensus.cc:2454] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83bf3b587b564a1d88581a50d7f2d66b in term 1.
I20260923 14:45:27.959123 31523 leader_election.cc:302] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dadb6be886d4e33b8ed2b5057ab2f5f, 83bf3b587b564a1d88581a50d7f2d66b; no voters:
I20260923 14:45:27.959389 31793 raft_consensus.cc:2781] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:27.959636 31793 raft_consensus.cc:686] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [term 1 LEADER]: Becoming Leader. State: Replica: 83bf3b587b564a1d88581a50d7f2d66b, State: Running, Role: LEADER
I20260923 14:45:27.960007 31793 consensus_queue.cc:237] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37679 } } peers { permanent_uuid: "83bf3b587b564a1d88581a50d7f2d66b" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41027 } } peers { permanent_uuid: "1e191f03734e4704b93f1262a9fe974a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40975 } }
I20260923 14:45:27.963253 31303 catalog_manager.cc:5495] T dbbd97a0da5d4207ab0cfe57393f5179 P 83bf3b587b564a1d88581a50d7f2d66b reported cstate change: term changed from 0 to 1, leader changed from <none> to 83bf3b587b564a1d88581a50d7f2d66b (127.30.2.194). New cstate: current_term: 1 leader_uuid: "83bf3b587b564a1d88581a50d7f2d66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dadb6be886d4e33b8ed2b5057ab2f5f" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83bf3b587b564a1d88581a50d7f2d66b" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e191f03734e4704b93f1262a9fe974a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40975 } health_report { overall_health: UNKNOWN } } }
Sep 23 14:45:27 dist-test-slave-wjf2 krb5kdc[31260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:45:27.987627 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:27.978279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.30.2.193:37679)
0923 14:45:27.978431 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:27.978436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:27.978569 (+ 133us) server_negotiation.cc:407] Connection header received
0923 14:45:27.978714 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:27.978718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:27.978768 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:27.978853 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:27.979542 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.980119 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.980960 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:27.981175 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:27.981281 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:27.983706 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:27.983719 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:27.983736 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:27.983761 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:27.985956 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.986358 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.986364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.986365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.986450 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:27.986683 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:27.986685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:27.986686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:27.986954 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:27.987121 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:27.987265 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 14:45:27.987389 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 23 14:45:27 dist-test-slave-wjf2 krb5kdc[31260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 14:45:28 dist-test-slave-wjf2 krb5kdc[31260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:45:28.014576 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:28.005889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.30.2.193:37679)
0923 14:45:28.006043 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:28.006048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:28.006197 (+ 149us) server_negotiation.cc:407] Connection header received
0923 14:45:28.006351 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:28.006354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:28.006402 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:28.006484 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:28.007151 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:28.007674 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:28.008504 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:28.008742 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:28.008850 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:28.011209 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:28.011213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:28.011215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:28.011240 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:28.012802 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:28.013237 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:28.013241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:28.013243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:28.013322 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:28.013601 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:28.013604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:28.013606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:28.013782 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:28.013926 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:28.014207 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 14:45:28.014333 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 14:45:28 dist-test-slave-wjf2 krb5kdc[31260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 14:45:28 dist-test-slave-wjf2 krb5kdc[31260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:45:28.040934 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:28.032342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.30.2.193:37679)
0923 14:45:28.032485 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:28.032491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:28.032639 (+ 148us) server_negotiation.cc:407] Connection header received
0923 14:45:28.032751 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:28.032754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:28.032805 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:28.032891 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:28.033563 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:28.034340 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:28.035266 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:28.035610 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:28.035729 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:28.038216 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:28.038219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:28.038222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:28.038246 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:28.039470 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:28.039894 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:28.039897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:28.039898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:28.039966 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:28.040213 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:28.040216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:28.040217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:28.040366 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:28.040461 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:28.040655 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 14:45:28.040781 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260923 14:45:28.042320 31412 tablet_service.cc:2908] Rejecting scan request for tablet dbbd97a0da5d4207ab0cfe57393f5179: Uninitialized: clean time has not yet been initialized
I20260923 14:45:28.044472 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 31368
W20260923 14:45:28.045655 31769 tablet.cc:2369] T dbbd97a0da5d4207ab0cfe57393f5179 P 1e191f03734e4704b93f1262a9fe974a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:45:28.051954 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 31504
I20260923 14:45:28.058274 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 31639
I20260923 14:45:28.063623 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 31273
2026-09-23T14:45:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2481 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:28 dist-test-slave-wjf2 krb5kdc[31810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:28 dist-test-slave-wjf2 krb5kdc[31810](info): set up 2 sockets
Sep 23 14:45:28 dist-test-slave-wjf2 krb5kdc[31810](info): commencing operation
krb5kdc: starting...
W20260923 14:45:29.109078 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:29 dist-test-slave-wjf2 krb5kdc[31810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:45:29.230926 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:35891
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:35891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:29.341917 31823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:29.342236 31823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:29.342311 31823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:29.346311 31823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:29.346391 31823 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 14:45:29.346426 31823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:29.346468 31823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:29.346501 31823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:29.346537 31823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:29.351764 31823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:35891
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:35891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:29.352872 31823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:29.353917 31823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:29.362365 31831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:29.362358 31829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:29.362416 31828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:29.362757 31823 server_base.cc:1034] running on GCE node
I20260923 14:45:29.363291 31823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:29.364468 31823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:29.365628 31823 hybrid_clock.cc:648] HybridClock initialized: now 1790174729365614 us; error 28 us; skew 500 ppm
Sep 23 14:45:29 dist-test-slave-wjf2 krb5kdc[31810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:29.374245 31823 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:45:29.375171 31823 webserver.cc:460] Webserver started at http://127.30.2.254:45717/ using document root <none> and password file <none>
I20260923 14:45:29.375757 31823 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:29.375871 31823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:29.376111 31823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:29.378188 31823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "904a83df8b024cea86adcde46992bafc"
format_stamp: "Formatted at 2026-09-23 14:45:29 on dist-test-slave-wjf2"
server_key: "6ec9beca0c96f8edb6adde9ea430344c"
server_key_iv: "83f76d6e7261880a46f17f1b942748b4"
server_key_version: "encryptionkey@0"
I20260923 14:45:29.378765 31823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "904a83df8b024cea86adcde46992bafc"
format_stamp: "Formatted at 2026-09-23 14:45:29 on dist-test-slave-wjf2"
server_key: "6ec9beca0c96f8edb6adde9ea430344c"
server_key_iv: "83f76d6e7261880a46f17f1b942748b4"
server_key_version: "encryptionkey@0"
I20260923 14:45:29.382666 31823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 14:45:29.385181 31838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:29.386386 31823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:45:29.386518 31823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "904a83df8b024cea86adcde46992bafc"
format_stamp: "Formatted at 2026-09-23 14:45:29 on dist-test-slave-wjf2"
server_key: "6ec9beca0c96f8edb6adde9ea430344c"
server_key_iv: "83f76d6e7261880a46f17f1b942748b4"
server_key_version: "encryptionkey@0"
I20260923 14:45:29.386619 31823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:29.403131 31823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:29.405934 31823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:29.406122 31823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:29.414026 31823 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:35891
I20260923 14:45:29.414052 31900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:35891 every 8 connection(s)
I20260923 14:45:29.414945 31823 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:29.416806 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 31823
I20260923 14:45:29.416898 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:29.417162 30731 external_mini_cluster.cc:1428] Setting key 44e394e026bcd2c79c87f4b48e1a1e66
I20260923 14:45:29.418114 31901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:29.425385 31901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc: Bootstrap starting.
Sep 23 14:45:29 dist-test-slave-wjf2 krb5kdc[31810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:29.428318 31901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:29.429159 31901 log.cc:824] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:29.431473 31901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc: No bootstrap required, opened a new log
I20260923 14:45:29.432724 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:29.418547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.30.2.254:35891)
0923 14:45:29.418982 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:29.419003 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:29.419051 (+ 48us) server_negotiation.cc:407] Connection header received
0923 14:45:29.419994 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:29.420019 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:29.420432 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:29.420824 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:29.421478 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.423026 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.423922 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.424652 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.424827 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:29.427588 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:29.427593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:29.427605 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:29.427632 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:29.429910 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:29.430426 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:29.430430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:29.430435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:29.430537 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:29.430842 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:29.430844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:29.430845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:29.431213 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:29.431353 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:29.431720 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 14:45:29.432013 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260923 14:45:29.434741 31901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904a83df8b024cea86adcde46992bafc" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35891 } }
I20260923 14:45:29.434999 31901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:29.435093 31901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904a83df8b024cea86adcde46992bafc, State: Initialized, Role: FOLLOWER
I20260923 14:45:29.435539 31901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904a83df8b024cea86adcde46992bafc" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35891 } }
I20260923 14:45:29.435691 31901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:29.435761 31901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:29.435889 31901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:29.436808 31901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904a83df8b024cea86adcde46992bafc" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35891 } }
I20260923 14:45:29.437163 31901 leader_election.cc:302] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 904a83df8b024cea86adcde46992bafc; no voters:
I20260923 14:45:29.437471 31901 leader_election.cc:288] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:29.437592 31906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:29.437850 31906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [term 1 LEADER]: Becoming Leader. State: Replica: 904a83df8b024cea86adcde46992bafc, State: Running, Role: LEADER
I20260923 14:45:29.438452 31906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904a83df8b024cea86adcde46992bafc" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35891 } }
I20260923 14:45:29.438716 31901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:29.440231 31907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "904a83df8b024cea86adcde46992bafc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904a83df8b024cea86adcde46992bafc" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35891 } } }
I20260923 14:45:29.440348 31907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:29.440285 31908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 904a83df8b024cea86adcde46992bafc. Latest consensus state: current_term: 1 leader_uuid: "904a83df8b024cea86adcde46992bafc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904a83df8b024cea86adcde46992bafc" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35891 } } }
I20260923 14:45:29.440403 31908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:29.440796 31917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:29.444281 31917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:29.449887 31917 catalog_manager.cc:1347] Generated new cluster ID: 57efb39418ee48c49e7501539bd81f46
I20260923 14:45:29.449958 31917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:29.454761 31917 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:29.455670 31917 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:29.463341 31917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc: Generated new TSK 0
I20260923 14:45:29.464087 31917 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:45:29.502269 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35891
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:29.630664 31928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:29.630900 31928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:29.630968 31928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:29.634982 31928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:29.635040 31928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:29.635078 31928 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:45:29.635174 31928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:29.640638 31928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.2.254:35891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:29.641683 31928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:29.642731 31928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:45:29.652200 31928 server_base.cc:1034] running on GCE node
W20260923 14:45:29.652153 31933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:29.652114 31934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:29.652112 31936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:29.652782 31928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:29.653409 31928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:29.654565 31928 hybrid_clock.cc:648] HybridClock initialized: now 1790174729654540 us; error 35 us; skew 500 ppm
Sep 23 14:45:29 dist-test-slave-wjf2 krb5kdc[31810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:29.657230 31928 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:45:29.658018 31928 webserver.cc:460] Webserver started at http://127.30.2.193:41507/ using document root <none> and password file <none>
I20260923 14:45:29.658550 31928 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:29.658635 31928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:29.658864 31928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:29.660862 31928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "44ba5786c8204f4aa3cfdd7457a711fc"
format_stamp: "Formatted at 2026-09-23 14:45:29 on dist-test-slave-wjf2"
server_key: "793427728e73830a06b45099d5010f16"
server_key_iv: "4ca5f96353808a764fd8d6805bcc0816"
server_key_version: "encryptionkey@0"
I20260923 14:45:29.661429 31928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "44ba5786c8204f4aa3cfdd7457a711fc"
format_stamp: "Formatted at 2026-09-23 14:45:29 on dist-test-slave-wjf2"
server_key: "793427728e73830a06b45099d5010f16"
server_key_iv: "4ca5f96353808a764fd8d6805bcc0816"
server_key_version: "encryptionkey@0"
I20260923 14:45:29.664997 31928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:45:29.667402 31943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:29.668845 31928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:45:29.668962 31928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "44ba5786c8204f4aa3cfdd7457a711fc"
format_stamp: "Formatted at 2026-09-23 14:45:29 on dist-test-slave-wjf2"
server_key: "793427728e73830a06b45099d5010f16"
server_key_iv: "4ca5f96353808a764fd8d6805bcc0816"
server_key_version: "encryptionkey@0"
I20260923 14:45:29.669061 31928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:29.695159 31928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:29.698414 31928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:29.698628 31928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:29.700304 31928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:29.700353 31928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:29.700397 31928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:29.700464 31928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:45:29 dist-test-slave-wjf2 krb5kdc[31810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:29.712744 31928 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:39789
I20260923 14:45:29.712808 32058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:39789 every 8 connection(s)
I20260923 14:45:29.713876 31928 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:45:29.715651 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:29.701798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35891 (local address 127.30.2.193:37349)
0923 14:45:29.702232 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:29.702280 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:29.703449 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:29.704111 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:29.704121 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:29.704651 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:29.705447 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:29.705474 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.706368 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.706372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:29.707261 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:29.707271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.707944 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.707948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:29.708157 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:29.708731 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:29.708751 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:29.710463 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:29.712657 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:29.712663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:29.712677 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:29.713086 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:29.713446 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:29.713449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:29.713450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:29.713595 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:29.714089 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:29.714125 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:29.714415 (+ 290us) client_negotiation.cc:769] Sending connection context
0923 14:45:29.714613 (+ 198us) client_negotiation.cc:240] Negotiation successful
0923 14:45:29.714931 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":125,"threads_started":1}
I20260923 14:45:29.716248 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:29.701017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:37349 (local address 127.30.2.254:35891)
0923 14:45:29.701188 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:29.701194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:29.702395 (+ 1201us) server_negotiation.cc:407] Connection header received
0923 14:45:29.703679 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:29.703682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:29.703739 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:29.703866 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:29.705646 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.706238 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.707387 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.707729 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.707895 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:29.710625 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:29.710629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:29.710632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:29.710663 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:29.712521 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:29.713200 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:29.713204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:29.713206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:29.713307 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:29.713696 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:29.713700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:29.713702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:29.713986 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:29.714084 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:29.715935 (+ 1851us) server_negotiation.cc:299] Negotiation successful
0923 14:45:29.716118 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 14:45:29.717325 32059 heartbeater.cc:347] Connected to a master server at 127.30.2.254:35891
I20260923 14:45:29.717499 32059 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:29.718118 32059 heartbeater.cc:511] Master 127.30.2.254:35891 requested a full tablet report, sending...
I20260923 14:45:29.719251 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 31928
I20260923 14:45:29.719377 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:29.719686 30731 external_mini_cluster.cc:1428] Setting key 531e0d58a459a9202c9e7ab3ff2b253c
I20260923 14:45:29.719923 31854 ts_manager.cc:194] Registered new tserver with Master: 44ba5786c8204f4aa3cfdd7457a711fc (127.30.2.193:39789)
I20260923 14:45:29.721691 31854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:37349
I20260923 14:45:29.728397 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:29.719668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:39305 (local address 127.30.2.254:35891)
0923 14:45:29.719809 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:29.719814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:29.719834 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:45:29.719874 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:29.719877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:29.719920 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:29.720008 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:29.720516 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.721560 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.722874 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.723201 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.723310 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:29.724781 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:29.724784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:29.724786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:29.724807 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:29.726440 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:29.727004 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:29.727006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:29.727007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:29.727081 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:29.727464 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:29.727467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:29.727469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:29.727660 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:29.727763 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:29.728170 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 14:45:29.728273 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":220,"server-negotiator.queue_time_us":52}
I20260923 14:45:29.728528 32065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:29.719359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35891 (local address 127.30.2.193:39305)
0923 14:45:29.719667 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:29.719682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:29.719770 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:29.720055 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:29.720061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:29.720286 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:29.720403 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:29.720409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.721736 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.721740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:29.722741 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:29.722757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.723340 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.723347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:29.723475 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:29.724108 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:29.724130 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:29.724619 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:29.726565 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:29.726572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:29.726576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:29.726893 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:29.727211 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:29.727217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:29.727220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:29.727336 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:29.727807 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:29.727811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:29.727941 (+ 130us) client_negotiation.cc:769] Sending connection context
0923 14:45:29.728092 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 14:45:29.728261 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:45:29.763284 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35891
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:29.874814 32068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:29.875052 32068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:29.875102 32068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:29.879109 32068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:29.879164 32068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:29.879201 32068 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:45:29.879232 32068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:29.884482 32068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.2.254:35891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:29.885550 32068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:29.886556 32068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:29.896041 32076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:29.896122 32068 server_base.cc:1034] running on GCE node
W20260923 14:45:29.896041 32073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:29.896059 32074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:29.896780 32068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:29.897470 32068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:29.898653 32068 hybrid_clock.cc:648] HybridClock initialized: now 1790174729898634 us; error 28 us; skew 500 ppm
Sep 23 14:45:29 dist-test-slave-wjf2 krb5kdc[31810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:29.901489 32068 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:45:29.902426 32068 webserver.cc:460] Webserver started at http://127.30.2.194:38993/ using document root <none> and password file <none>
I20260923 14:45:29.902988 32068 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:29.903118 32068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:29.903348 32068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:29.905460 32068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "a076f8d22b5e4d59b34e963542dd1b20"
format_stamp: "Formatted at 2026-09-23 14:45:29 on dist-test-slave-wjf2"
server_key: "82a04a13c5cb92fe3fc9d251a3c116fe"
server_key_iv: "71ef0a65be5b47fae022756497a4161f"
server_key_version: "encryptionkey@0"
I20260923 14:45:29.906031 32068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "a076f8d22b5e4d59b34e963542dd1b20"
format_stamp: "Formatted at 2026-09-23 14:45:29 on dist-test-slave-wjf2"
server_key: "82a04a13c5cb92fe3fc9d251a3c116fe"
server_key_iv: "71ef0a65be5b47fae022756497a4161f"
server_key_version: "encryptionkey@0"
I20260923 14:45:29.909847 32068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:45:29.912343 32083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:29.913584 32068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:29.913723 32068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "a076f8d22b5e4d59b34e963542dd1b20"
format_stamp: "Formatted at 2026-09-23 14:45:29 on dist-test-slave-wjf2"
server_key: "82a04a13c5cb92fe3fc9d251a3c116fe"
server_key_iv: "71ef0a65be5b47fae022756497a4161f"
server_key_version: "encryptionkey@0"
I20260923 14:45:29.913825 32068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:29.936730 32068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:29.940613 32068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:29.940798 32068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:29.942194 32068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:29.942261 32068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:29.942324 32068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:29.942353 32068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:45:29 dist-test-slave-wjf2 krb5kdc[31810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:29.952998 32068 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:41983
I20260923 14:45:29.953065 32198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:41983 every 8 connection(s)
I20260923 14:45:29.954085 32068 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:29.955513 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:29.943047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:55417 (local address 127.30.2.254:35891)
0923 14:45:29.943191 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:29.943195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:29.944082 (+ 887us) server_negotiation.cc:407] Connection header received
0923 14:45:29.945071 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:29.945074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:29.945141 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:29.945238 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:29.946766 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.947305 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.948613 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.948844 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.948953 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:29.951680 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:29.951686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:29.951690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:29.951720 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:29.953427 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:29.953892 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:29.953896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:29.953898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:29.953996 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:29.954398 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:29.954404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:29.954409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:29.954665 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:29.954773 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:29.955302 (+ 529us) server_negotiation.cc:299] Negotiation successful
0923 14:45:29.955399 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:45:29.956339 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:29.943451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35891 (local address 127.30.2.194:55417)
0923 14:45:29.943926 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:29.943969 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:29.944796 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:29.945559 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:29.945567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:29.946022 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:29.946580 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:29.946602 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.947429 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.947433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:29.948488 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:29.948498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.949005 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.949009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:29.949194 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:29.949827 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:29.949846 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:29.951493 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:29.953537 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:29.953541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:29.953552 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:29.953790 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:29.954089 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:29.954091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:29.954131 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:29.954305 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:29.954831 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:29.954840 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:29.955090 (+ 250us) client_negotiation.cc:769] Sending connection context
0923 14:45:29.955245 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 14:45:29.955468 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":134,"threads_started":1}
I20260923 14:45:29.957391 32199 heartbeater.cc:347] Connected to a master server at 127.30.2.254:35891
I20260923 14:45:29.957576 32199 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:29.958094 32199 heartbeater.cc:511] Master 127.30.2.254:35891 requested a full tablet report, sending...
I20260923 14:45:29.959158 31855 ts_manager.cc:194] Registered new tserver with Master: a076f8d22b5e4d59b34e963542dd1b20 (127.30.2.194:41983)
I20260923 14:45:29.959901 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 32068
I20260923 14:45:29.959931 31855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:55417
I20260923 14:45:29.960011 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:29.960342 30731 external_mini_cluster.cc:1428] Setting key a88a6039efe1b8d415e3f87b89eb3cd4
I20260923 14:45:29.968564 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:29.959526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:43773 (local address 127.30.2.254:35891)
0923 14:45:29.959632 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:29.959641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:29.960867 (+ 1226us) server_negotiation.cc:407] Connection header received
0923 14:45:29.961046 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:29.961049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:29.961090 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:29.961181 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:29.961794 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.962428 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.963672 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.964061 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.964221 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:29.965366 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:29.965369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:29.965371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:29.965391 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:29.966837 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:29.967356 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:29.967358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:29.967359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:29.967428 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:29.967737 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:29.967739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:29.967740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:29.967993 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:29.968075 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:29.968375 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 14:45:29.968459 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:45:29.968698 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:29.959463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35891 (local address 127.30.2.194:43773)
0923 14:45:29.960776 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:29.960789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:29.960935 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:29.961196 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:29.961199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:29.961437 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:29.961656 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:29.961663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.962563 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.962570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:29.963536 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:29.963546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:29.964212 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:29.964215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:29.964283 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:29.964783 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:29.964801 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:29.965240 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:29.966967 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:29.966973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:29.966977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:29.967255 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:29.967519 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:29.967523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:29.967528 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:29.967646 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:29.968092 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:29.968095 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:29.968206 (+ 111us) client_negotiation.cc:769] Sending connection context
0923 14:45:29.968326 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 14:45:29.968466 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"mutex_wait_us":21,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:45:29.987133 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35891
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:30.098285 32208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:30.098516 32208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:30.098579 32208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:30.102537 32208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:30.102586 32208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:30.102658 32208 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:45:30.102730 32208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:30.107910 32208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.2.254:35891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:30.108865 32208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:30.109845 32208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:30.119122 32217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:30.119127 32214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:30.119201 32213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:30.119589 32208 server_base.cc:1034] running on GCE node
I20260923 14:45:30.120036 32208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:30.120635 32208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:30.121789 32208 hybrid_clock.cc:648] HybridClock initialized: now 1790174730121775 us; error 27 us; skew 500 ppm
Sep 23 14:45:30 dist-test-slave-wjf2 krb5kdc[31810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174730, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:30.124596 32208 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:45:30.125443 32208 webserver.cc:460] Webserver started at http://127.30.2.195:35709/ using document root <none> and password file <none>
I20260923 14:45:30.126008 32208 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:30.126129 32208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:30.126355 32208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:30.128394 32208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "42a4fc3c45004ef0970ca8f332753c82"
format_stamp: "Formatted at 2026-09-23 14:45:30 on dist-test-slave-wjf2"
server_key: "88f9fd70a164f4bf8c1b44987d324501"
server_key_iv: "8d99951b8822e5b91a5e73dae6ea173d"
server_key_version: "encryptionkey@0"
I20260923 14:45:30.128931 32208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "42a4fc3c45004ef0970ca8f332753c82"
format_stamp: "Formatted at 2026-09-23 14:45:30 on dist-test-slave-wjf2"
server_key: "88f9fd70a164f4bf8c1b44987d324501"
server_key_iv: "8d99951b8822e5b91a5e73dae6ea173d"
server_key_version: "encryptionkey@0"
I20260923 14:45:30.132558 32208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:45:30.134876 32225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:30.136072 32208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260923 14:45:30.136194 32208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "42a4fc3c45004ef0970ca8f332753c82"
format_stamp: "Formatted at 2026-09-23 14:45:30 on dist-test-slave-wjf2"
server_key: "88f9fd70a164f4bf8c1b44987d324501"
server_key_iv: "8d99951b8822e5b91a5e73dae6ea173d"
server_key_version: "encryptionkey@0"
I20260923 14:45:30.136291 32208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:30.156409 32208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:30.159602 32208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:30.159799 32208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:30.161218 32208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:30.161285 32208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:30.161351 32208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:30.161391 32208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:45:30 dist-test-slave-wjf2 krb5kdc[31810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174730, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:30.172524 32208 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:44945
I20260923 14:45:30.173369 32208 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:45:30.173484 32340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:44945 every 8 connection(s)
I20260923 14:45:30.175426 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.162024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:47591 (local address 127.30.2.254:35891)
0923 14:45:30.162178 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.162183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.163151 (+ 968us) server_negotiation.cc:407] Connection header received
0923 14:45:30.164344 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.164347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.164407 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.164513 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:30.165796 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.166377 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.167798 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.168165 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.168298 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.171330 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:30.171335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:30.171338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:30.171366 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:30.173255 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:30.173970 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:30.173974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:30.173976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:30.174069 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:30.174492 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:30.174495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:30.174496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:30.174674 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:30.174762 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.175216 (+ 454us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.175310 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:45:30.176381 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.162477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35891 (local address 127.30.2.195:47591)
0923 14:45:30.162978 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.163030 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.164119 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.164720 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.164731 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.165120 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:30.165649 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.165669 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.166515 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.166519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.167648 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.167663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.168342 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.168347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.168524 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.169172 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:30.169200 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:30.171131 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:30.173374 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:30.173381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:30.173396 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:30.173783 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:30.174266 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:30.174268 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:30.174270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:30.174396 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:30.174770 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:30.174776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:30.174987 (+ 211us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.175176 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.175414 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260923 14:45:30.177716 32341 heartbeater.cc:347] Connected to a master server at 127.30.2.254:35891
I20260923 14:45:30.177884 32341 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:30.178372 32341 heartbeater.cc:511] Master 127.30.2.254:35891 requested a full tablet report, sending...
I20260923 14:45:30.179482 31855 ts_manager.cc:194] Registered new tserver with Master: 42a4fc3c45004ef0970ca8f332753c82 (127.30.2.195:44945)
I20260923 14:45:30.180389 31855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:47591
I20260923 14:45:30.183103 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 32208
I20260923 14:45:30.183262 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:45:30.183559 30731 external_mini_cluster.cc:1428] Setting key a2d3d75a8b4ede95a6316eb257186f2b
I20260923 14:45:30.185997 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:45:30.187815 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.179939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:49289 (local address 127.30.2.254:35891)
0923 14:45:30.180071 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.180075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.180283 (+ 208us) server_negotiation.cc:407] Connection header received
0923 14:45:30.180377 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.180379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.180415 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.180484 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:30.180993 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.181516 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.182662 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.183019 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.183156 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.184476 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:30.184478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:30.184480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:30.184500 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:30.186159 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:30.186609 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:30.186612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:30.186613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:30.186678 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:30.187003 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:30.187005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:30.187006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:30.187176 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:30.187251 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.187583 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.187672 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:45:30.188050 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.179874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35891 (local address 127.30.2.195:49289)
0923 14:45:30.180168 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.180183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.180293 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.180566 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.180569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.180770 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:30.180862 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.180868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.181643 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.181647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.182529 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.182539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.183193 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.183198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.183328 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.183904 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:30.183929 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:30.184347 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:30.186256 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:30.186259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:30.186261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:30.186510 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:30.186766 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:30.186771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:30.186774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:30.186913 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:30.187272 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:30.187278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:30.187401 (+ 123us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.187521 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.187690 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":112,"threads_started":1}
Sep 23 14:45:30 dist-test-slave-wjf2 krb5kdc[31810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174730, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 14:45:30 dist-test-slave-wjf2 krb5kdc[31810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174730, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:30.212396 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.203934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.30.2.254:35891)
0923 14:45:30.204125 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.204128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.204144 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:45:30.204185 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.204187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.204229 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.204346 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:30.205095 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.205640 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.206506 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.206760 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.206880 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.209293 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:30.209298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:30.209300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:30.209325 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:30.210571 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:30.211028 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:30.211033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:30.211035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:30.211130 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:30.211436 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:30.211440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:30.211441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:30.211634 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:30.211780 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.212082 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.212212 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 14:45:30.215327 31855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:45:30.218602 31855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:30.233282 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.227998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.30.2.195:44945)
0923 14:45:30.228428 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.228441 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.228464 (+ 23us) server_negotiation.cc:407] Connection header received
0923 14:45:30.228543 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.228549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.228759 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.229185 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:30.229329 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.230229 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.231956 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.232601 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.232730 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.232917 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.233061 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.233150 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":112,"threads_started":1}
I20260923 14:45:30.233278 32358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.227264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41983 (local address 127.0.0.1:33450)
0923 14:45:30.227747 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.227817 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.227949 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.228957 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.228963 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.229006 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:30.229179 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.229187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.230361 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.230366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.231778 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.231812 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.232753 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.232756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.232855 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.232900 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.232981 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.233101 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":101,"spinlock_wait_cycles":2560,"thread_start_us":78,"threads_started":1}
I20260923 14:45:30.233275 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.227533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:44945 (local address 127.0.0.1:54936)
0923 14:45:30.228061 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.228079 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.228188 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.228957 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.228964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.229002 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:30.229220 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.229230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.230391 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.230394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.231819 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.231830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.232732 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.232735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.232852 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.232891 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.233029 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.233105 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":83,"spinlock_wait_cycles":6144,"thread_start_us":88,"threads_started":1}
I20260923 14:45:30.233762 32361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.228018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:39789 (local address 127.0.0.1:54494)
0923 14:45:30.228868 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.228880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.229018 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.229656 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.229659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.229676 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:30.229748 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.229753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.231062 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.231065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.232210 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.232219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.233401 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.233405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.233491 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.233516 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.233587 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.233648 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":94,"threads_started":1}
I20260923 14:45:30.233774 32359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.227439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33450 (local address 127.30.2.194:41983)
0923 14:45:30.228428 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.228441 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.228464 (+ 23us) server_negotiation.cc:407] Connection header received
0923 14:45:30.228543 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.228549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.228759 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.228960 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:30.229307 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.230240 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.231925 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.232613 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.233342 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.233489 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.233536 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.233641 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":123,"threads_started":1}
I20260923 14:45:30.234234 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.228336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.30.2.193:39789)
0923 14:45:30.228719 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.228726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.228958 (+ 232us) server_negotiation.cc:407] Connection header received
0923 14:45:30.229179 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.229185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.229388 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.229499 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:30.229868 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.230930 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.232335 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.233111 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.233210 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.233339 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.233985 (+ 646us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.234090 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":82,"threads_started":1}
I20260923 14:45:30.236995 32133 tablet_service.cc:1467] Processing CreateTablet for tablet 204cd82ae052457f840bec4236e04220 (DEFAULT_TABLE table=test-table [id=b33accaae94240b7a0706144ba1e0ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:30.237334 31993 tablet_service.cc:1467] Processing CreateTablet for tablet 204cd82ae052457f840bec4236e04220 (DEFAULT_TABLE table=test-table [id=b33accaae94240b7a0706144ba1e0ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:30.237426 32275 tablet_service.cc:1467] Processing CreateTablet for tablet 204cd82ae052457f840bec4236e04220 (DEFAULT_TABLE table=test-table [id=b33accaae94240b7a0706144ba1e0ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:30.238149 32133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204cd82ae052457f840bec4236e04220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:30.238350 31993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204cd82ae052457f840bec4236e04220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:30.238413 32275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204cd82ae052457f840bec4236e04220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:30.244549 32364 tablet_bootstrap.cc:492] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc: Bootstrap starting.
I20260923 14:45:30.246140 32365 tablet_bootstrap.cc:492] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20: Bootstrap starting.
I20260923 14:45:30.246958 32364 tablet_bootstrap.cc:654] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:30.247227 32366 tablet_bootstrap.cc:492] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: Bootstrap starting.
I20260923 14:45:30.247903 32364 log.cc:824] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:30.248443 32365 tablet_bootstrap.cc:654] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:30.249086 32366 tablet_bootstrap.cc:654] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:30.249344 32365 log.cc:824] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:30.249974 32364 tablet_bootstrap.cc:492] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc: No bootstrap required, opened a new log
I20260923 14:45:30.250160 32366 log.cc:824] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:30.250249 32364 ts_tablet_manager.cc:1397] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:45:30.251703 32366 tablet_bootstrap.cc:492] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: No bootstrap required, opened a new log
I20260923 14:45:30.251911 32366 ts_tablet_manager.cc:1397] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:45:30.251919 32365 tablet_bootstrap.cc:492] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20: No bootstrap required, opened a new log
I20260923 14:45:30.252089 32365 ts_tablet_manager.cc:1397] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:45:30.254678 32366 raft_consensus.cc:348] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.254904 32366 raft_consensus.cc:374] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:30.254983 32366 raft_consensus.cc:729] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a4fc3c45004ef0970ca8f332753c82, State: Initialized, Role: FOLLOWER
I20260923 14:45:30.254869 32364 raft_consensus.cc:348] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.255108 32364 raft_consensus.cc:374] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:30.255232 32364 raft_consensus.cc:729] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ba5786c8204f4aa3cfdd7457a711fc, State: Initialized, Role: FOLLOWER
I20260923 14:45:30.255477 32366 consensus_queue.cc:260] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.255631 32364 consensus_queue.cc:260] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.255856 32365 raft_consensus.cc:348] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.256132 32365 raft_consensus.cc:374] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:30.256212 32365 raft_consensus.cc:729] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a076f8d22b5e4d59b34e963542dd1b20, State: Initialized, Role: FOLLOWER
I20260923 14:45:30.256354 32059 heartbeater.cc:503] Master 127.30.2.254:35891 was elected leader, sending a full tablet report...
I20260923 14:45:30.256513 32366 ts_tablet_manager.cc:1428] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:45:30.256598 32364 ts_tablet_manager.cc:1428] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 14:45:30.256652 32365 consensus_queue.cc:260] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.257087 32341 heartbeater.cc:503] Master 127.30.2.254:35891 was elected leader, sending a full tablet report...
I20260923 14:45:30.257591 32365 ts_tablet_manager.cc:1428] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:45:30.257779 32199 heartbeater.cc:503] Master 127.30.2.254:35891 was elected leader, sending a full tablet report...
I20260923 14:45:30.388187 32372 raft_consensus.cc:482] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:30.388387 32372 raft_consensus.cc:504] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.389489 32372 leader_election.cc:288] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44ba5786c8204f4aa3cfdd7457a711fc (127.30.2.193:39789), 42a4fc3c45004ef0970ca8f332753c82 (127.30.2.195:44945)
I20260923 14:45:30.393486 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.389760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:44945 (local address 127.30.2.194:52433)
0923 14:45:30.389966 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.389982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.390081 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.390435 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.390438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.390451 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:30.390564 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.390572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.391394 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.391397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.392376 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.392389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.393035 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.393041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.393182 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.393221 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.393302 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.393378 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260923 14:45:30.393543 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.389899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:52433 (local address 127.30.2.195:44945)
0923 14:45:30.390049 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.390054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.390074 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:45:30.390236 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.390239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.390292 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.390401 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:30.390697 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.391281 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.392517 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.392913 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.393050 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.393114 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.393331 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.393404 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 14:45:30.394207 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.389768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:39789 (local address 127.30.2.194:39347)
0923 14:45:30.390246 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.390257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.390331 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.390586 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.390590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.390608 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:30.391202 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.391212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.392015 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.392019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.392925 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.392933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.393785 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.393789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.393894 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.393918 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.393998 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.394059 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":52,"thread_start_us":149,"threads_started":1}
I20260923 14:45:30.394210 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.389830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:39347 (local address 127.30.2.193:39789)
0923 14:45:30.389996 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.390003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.390335 (+ 332us) server_negotiation.cc:407] Connection header received
0923 14:45:30.390426 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.390429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.390484 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.390569 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:30.391328 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.391903 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.393077 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.393664 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.393794 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.393845 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.394022 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.394119 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 14:45:30.394191 32295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204cd82ae052457f840bec4236e04220" candidate_uuid: "a076f8d22b5e4d59b34e963542dd1b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a4fc3c45004ef0970ca8f332753c82" is_pre_election: true
I20260923 14:45:30.394510 32295 raft_consensus.cc:2454] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a076f8d22b5e4d59b34e963542dd1b20 in term 0.
I20260923 14:45:30.394824 32013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204cd82ae052457f840bec4236e04220" candidate_uuid: "a076f8d22b5e4d59b34e963542dd1b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" is_pre_election: true
I20260923 14:45:30.395045 32085 leader_election.cc:302] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a4fc3c45004ef0970ca8f332753c82, a076f8d22b5e4d59b34e963542dd1b20; no voters:
I20260923 14:45:30.395124 32013 raft_consensus.cc:2454] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a076f8d22b5e4d59b34e963542dd1b20 in term 0.
I20260923 14:45:30.395331 32372 raft_consensus.cc:2781] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:30.395473 32372 raft_consensus.cc:482] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:30.395548 32372 raft_consensus.cc:3037] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:30.396759 32372 raft_consensus.cc:504] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.397132 32372 leader_election.cc:288] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [CANDIDATE]: Term 1 election: Requested vote from peers 44ba5786c8204f4aa3cfdd7457a711fc (127.30.2.193:39789), 42a4fc3c45004ef0970ca8f332753c82 (127.30.2.195:44945)
I20260923 14:45:30.397466 32013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204cd82ae052457f840bec4236e04220" candidate_uuid: "a076f8d22b5e4d59b34e963542dd1b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ba5786c8204f4aa3cfdd7457a711fc"
I20260923 14:45:30.397466 32295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204cd82ae052457f840bec4236e04220" candidate_uuid: "a076f8d22b5e4d59b34e963542dd1b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a4fc3c45004ef0970ca8f332753c82"
I20260923 14:45:30.397619 32013 raft_consensus.cc:3037] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:30.397619 32295 raft_consensus.cc:3037] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:30.399017 32013 raft_consensus.cc:2454] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a076f8d22b5e4d59b34e963542dd1b20 in term 1.
I20260923 14:45:30.399348 32087 leader_election.cc:302] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ba5786c8204f4aa3cfdd7457a711fc, a076f8d22b5e4d59b34e963542dd1b20; no voters:
I20260923 14:45:30.399370 32295 raft_consensus.cc:2454] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a076f8d22b5e4d59b34e963542dd1b20 in term 1.
I20260923 14:45:30.399536 32372 raft_consensus.cc:2781] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:30.399773 32372 raft_consensus.cc:686] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 1 LEADER]: Becoming Leader. State: Replica: a076f8d22b5e4d59b34e963542dd1b20, State: Running, Role: LEADER
I20260923 14:45:30.400142 32372 consensus_queue.cc:237] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.403417 31855 catalog_manager.cc:5495] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to a076f8d22b5e4d59b34e963542dd1b20 (127.30.2.194). New cstate: current_term: 1 leader_uuid: "a076f8d22b5e4d59b34e963542dd1b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } health_report { overall_health: UNKNOWN } } }
W20260923 14:45:30.424435 32342 tablet.cc:2369] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:45:30.428891 32359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.424372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.30.2.194:41983)
0923 14:45:30.424511 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.424518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.424706 (+ 188us) server_negotiation.cc:407] Connection header received
0923 14:45:30.424924 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.424928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.424997 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.425136 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:30.425451 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.425957 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.427294 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.427523 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.427611 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.428041 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:30.428420 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:30.428505 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.428661 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.428724 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:45:30.434556 32013 raft_consensus.cc:1261] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc [term 1 FOLLOWER]: Refusing update from remote peer a076f8d22b5e4d59b34e963542dd1b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:30.434556 32295 raft_consensus.cc:1261] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82 [term 1 FOLLOWER]: Refusing update from remote peer a076f8d22b5e4d59b34e963542dd1b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:30.435292 32374 consensus_queue.cc:1035] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:30.437754 32374 consensus_queue.cc:1035] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:30.442750 32378 mvcc.cc:205] Tried to move back new op lower bound from 7332555695852814336 to 7332555695721709568. Current Snapshot: MvccSnapshot[applied={T|T < 7332555695852814336}]
I20260923 14:45:30.444422 32380 mvcc.cc:205] Tried to move back new op lower bound from 7332555695852814336 to 7332555695721709568. Current Snapshot: MvccSnapshot[applied={T|T < 7332555695852814336}]
I20260923 14:45:30.454272 31855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:30.454452 31855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:30.457120 31855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc: Sending DeleteTablet for 3 replicas of tablet 204cd82ae052457f840bec4236e04220
I20260923 14:45:30.458029 32275 tablet_service.cc:1514] Processing DeleteTablet for tablet 204cd82ae052457f840bec4236e04220 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:30 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:54936
I20260923 14:45:30.458106 32133 tablet_service.cc:1514] Processing DeleteTablet for tablet 204cd82ae052457f840bec4236e04220 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:30 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:33450
I20260923 14:45:30.458604 31993 tablet_service.cc:1514] Processing DeleteTablet for tablet 204cd82ae052457f840bec4236e04220 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:30 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:54494
I20260923 14:45:30.458981 32386 tablet_replica.cc:330] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: stopping tablet replica
I20260923 14:45:30.459041 32388 tablet_replica.cc:330] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc: stopping tablet replica
I20260923 14:45:30.458988 31855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 14:45:30.459295 32386 raft_consensus.cc:2229] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:30.459335 32388 raft_consensus.cc:2229] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc [term 1 FOLLOWER]: Raft consensus shutting down.
W20260923 14:45:30.459426 31855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:30.459609 32388 raft_consensus.cc:2258] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:30.459640 32386 raft_consensus.cc:2258] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:30.460047 32387 tablet_replica.cc:330] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20: stopping tablet replica
I20260923 14:45:30.460320 32387 raft_consensus.cc:2229] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:45:30.460732 32387 raft_consensus.cc:2258] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:30.461556 32386 ts_tablet_manager.cc:1905] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:30.462383 32388 ts_tablet_manager.cc:1905] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:30.464278 32386 ts_tablet_manager.cc:1918] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:45:30.464377 32386 log.cc:1196] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/204cd82ae052457f840bec4236e04220
I20260923 14:45:30.464699 32386 ts_tablet_manager.cc:1939] T 204cd82ae052457f840bec4236e04220 P 42a4fc3c45004ef0970ca8f332753c82: Deleting consensus metadata
I20260923 14:45:30.465406 32388 ts_tablet_manager.cc:1918] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:45:30.465541 32388 log.cc:1196] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/204cd82ae052457f840bec4236e04220
I20260923 14:45:30.465924 32388 ts_tablet_manager.cc:1939] T 204cd82ae052457f840bec4236e04220 P 44ba5786c8204f4aa3cfdd7457a711fc: Deleting consensus metadata
I20260923 14:45:30.465926 31840 catalog_manager.cc:4841] TS 42a4fc3c45004ef0970ca8f332753c82 (127.30.2.195:44945): tablet 204cd82ae052457f840bec4236e04220 (table test-table [id=b33accaae94240b7a0706144ba1e0ea6]) successfully deleted
I20260923 14:45:30.466892 32275 tablet_service.cc:1467] Processing CreateTablet for tablet ff377f25eb33468a9543f3d9a8d312fe (DEFAULT_TABLE table=test-table [id=05cca5002ddb4c78bb3a4a5266e3bbab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:30.467016 32387 ts_tablet_manager.cc:1905] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:30.467204 32275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff377f25eb33468a9543f3d9a8d312fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:30.467679 31842 catalog_manager.cc:4841] TS 44ba5786c8204f4aa3cfdd7457a711fc (127.30.2.193:39789): tablet 204cd82ae052457f840bec4236e04220 (table test-table [id=b33accaae94240b7a0706144ba1e0ea6]) successfully deleted
I20260923 14:45:30.468370 31993 tablet_service.cc:1467] Processing CreateTablet for tablet ff377f25eb33468a9543f3d9a8d312fe (DEFAULT_TABLE table=test-table [id=05cca5002ddb4c78bb3a4a5266e3bbab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:30.468608 31993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff377f25eb33468a9543f3d9a8d312fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:30.469135 32133 tablet_service.cc:1467] Processing CreateTablet for tablet ff377f25eb33468a9543f3d9a8d312fe (DEFAULT_TABLE table=test-table [id=05cca5002ddb4c78bb3a4a5266e3bbab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:30.469417 32133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff377f25eb33468a9543f3d9a8d312fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:30.469802 32366 tablet_bootstrap.cc:492] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82: Bootstrap starting.
I20260923 14:45:30.470680 32364 tablet_bootstrap.cc:492] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc: Bootstrap starting.
I20260923 14:45:30.470881 32366 tablet_bootstrap.cc:654] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:30.471655 32364 tablet_bootstrap.cc:654] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:30.472010 32365 tablet_bootstrap.cc:492] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20: Bootstrap starting.
I20260923 14:45:30.472604 32366 tablet_bootstrap.cc:492] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82: No bootstrap required, opened a new log
I20260923 14:45:30.472703 32366 ts_tablet_manager.cc:1397] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 14:45:30.472906 32365 tablet_bootstrap.cc:654] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:30.473289 32366 raft_consensus.cc:348] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.473423 32366 raft_consensus.cc:374] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:30.473464 32366 raft_consensus.cc:729] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a4fc3c45004ef0970ca8f332753c82, State: Initialized, Role: FOLLOWER
I20260923 14:45:30.473477 32364 tablet_bootstrap.cc:492] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc: No bootstrap required, opened a new log
I20260923 14:45:30.473551 32364 ts_tablet_manager.cc:1397] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260923 14:45:30.473601 32366 consensus_queue.cc:260] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.473901 32366 ts_tablet_manager.cc:1428] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 14:45:30.474275 32387 ts_tablet_manager.cc:1918] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:45:30.474268 32364 raft_consensus.cc:348] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.474372 32387 log.cc:1196] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/204cd82ae052457f840bec4236e04220
I20260923 14:45:30.474385 32364 raft_consensus.cc:374] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:30.474433 32364 raft_consensus.cc:729] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ba5786c8204f4aa3cfdd7457a711fc, State: Initialized, Role: FOLLOWER
I20260923 14:45:30.474579 32364 consensus_queue.cc:260] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.474704 32387 ts_tablet_manager.cc:1939] T 204cd82ae052457f840bec4236e04220 P a076f8d22b5e4d59b34e963542dd1b20: Deleting consensus metadata
I20260923 14:45:30.474905 32364 ts_tablet_manager.cc:1428] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260923 14:45:30.475486 32365 tablet_bootstrap.cc:492] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20: No bootstrap required, opened a new log
I20260923 14:45:30.475575 32365 ts_tablet_manager.cc:1397] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 14:45:30.475982 32365 raft_consensus.cc:348] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.476086 32365 raft_consensus.cc:374] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:30.476130 32365 raft_consensus.cc:729] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a076f8d22b5e4d59b34e963542dd1b20, State: Initialized, Role: FOLLOWER
I20260923 14:45:30.476243 32365 consensus_queue.cc:260] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.476471 32365 ts_tablet_manager.cc:1428] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:30.476457 31842 catalog_manager.cc:4841] TS a076f8d22b5e4d59b34e963542dd1b20 (127.30.2.194:41983): tablet 204cd82ae052457f840bec4236e04220 (table test-table [id=b33accaae94240b7a0706144ba1e0ea6]) successfully deleted
I20260923 14:45:30.493508 32370 raft_consensus.cc:482] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:30.493628 32370 raft_consensus.cc:504] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.494663 32370 leader_election.cc:288] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44ba5786c8204f4aa3cfdd7457a711fc (127.30.2.193:39789), a076f8d22b5e4d59b34e963542dd1b20 (127.30.2.194:41983)
I20260923 14:45:30.498821 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.494999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:57001 (local address 127.30.2.193:39789)
0923 14:45:30.495152 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.495158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.495215 (+ 57us) server_negotiation.cc:407] Connection header received
0923 14:45:30.495412 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.495416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.495463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.495541 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:30.495948 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.496687 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.497854 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.498257 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.498411 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.498463 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.498643 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.498714 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:45:30.498840 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.494898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:39789 (local address 127.30.2.195:57001)
0923 14:45:30.495151 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.495167 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.495263 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.495669 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.495671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.495688 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:30.495823 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.495829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.496799 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.496802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.497718 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.497728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.498403 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.498407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.498506 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.498532 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.498621 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.498694 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":1}
I20260923 14:45:30.499528 32013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff377f25eb33468a9543f3d9a8d312fe" candidate_uuid: "42a4fc3c45004ef0970ca8f332753c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" is_pre_election: true
I20260923 14:45:30.499679 32013 raft_consensus.cc:2454] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42a4fc3c45004ef0970ca8f332753c82 in term 0.
I20260923 14:45:30.499759 32359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.495521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:55365 (local address 127.30.2.194:41983)
0923 14:45:30.495672 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.495677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.495693 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:45:30.495748 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.495751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.495811 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.495897 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:30.496174 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.496894 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.498486 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.499112 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.499213 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.499268 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.499595 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.499664 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:45:30.499765 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.495164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41983 (local address 127.30.2.195:55365)
0923 14:45:30.495465 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.495478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.495556 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.495906 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.495912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.495930 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:30.496048 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.496053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.497040 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.497044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.498308 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.498321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.499355 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.499358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.499462 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.499487 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.499565 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.499638 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260923 14:45:30.500223 32229 leader_election.cc:302] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a4fc3c45004ef0970ca8f332753c82, 44ba5786c8204f4aa3cfdd7457a711fc; no voters:
I20260923 14:45:30.500267 32153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff377f25eb33468a9543f3d9a8d312fe" candidate_uuid: "42a4fc3c45004ef0970ca8f332753c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a076f8d22b5e4d59b34e963542dd1b20" is_pre_election: true
I20260923 14:45:30.500489 32370 raft_consensus.cc:2781] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:30.500499 32153 raft_consensus.cc:2454] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42a4fc3c45004ef0970ca8f332753c82 in term 0.
I20260923 14:45:30.500563 32370 raft_consensus.cc:482] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:30.500622 32370 raft_consensus.cc:3037] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:30.501740 32370 raft_consensus.cc:504] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.502082 32370 leader_election.cc:288] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [CANDIDATE]: Term 1 election: Requested vote from peers 44ba5786c8204f4aa3cfdd7457a711fc (127.30.2.193:39789), a076f8d22b5e4d59b34e963542dd1b20 (127.30.2.194:41983)
I20260923 14:45:30.502429 32013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff377f25eb33468a9543f3d9a8d312fe" candidate_uuid: "42a4fc3c45004ef0970ca8f332753c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ba5786c8204f4aa3cfdd7457a711fc"
I20260923 14:45:30.502463 32153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff377f25eb33468a9543f3d9a8d312fe" candidate_uuid: "42a4fc3c45004ef0970ca8f332753c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a076f8d22b5e4d59b34e963542dd1b20"
I20260923 14:45:30.502554 32153 raft_consensus.cc:3037] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:30.502549 32013 raft_consensus.cc:3037] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:30.503690 32153 raft_consensus.cc:2454] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42a4fc3c45004ef0970ca8f332753c82 in term 1.
I20260923 14:45:30.503690 32013 raft_consensus.cc:2454] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42a4fc3c45004ef0970ca8f332753c82 in term 1.
I20260923 14:45:30.504024 32229 leader_election.cc:302] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a4fc3c45004ef0970ca8f332753c82, 44ba5786c8204f4aa3cfdd7457a711fc; no voters:
I20260923 14:45:30.504303 32370 raft_consensus.cc:2781] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:30.504498 32370 raft_consensus.cc:686] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 1 LEADER]: Becoming Leader. State: Replica: 42a4fc3c45004ef0970ca8f332753c82, State: Running, Role: LEADER
I20260923 14:45:30.504731 32370 consensus_queue.cc:237] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.507104 31853 catalog_manager.cc:5495] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42a4fc3c45004ef0970ca8f332753c82 (127.30.2.195). New cstate: current_term: 1 leader_uuid: "42a4fc3c45004ef0970ca8f332753c82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } health_report { overall_health: HEALTHY } } }
Sep 23 14:45:30 dist-test-slave-wjf2 krb5kdc[31810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174729, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:30.547266 32400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.537936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35891 (local address 127.0.0.1:51276)
0923 14:45:30.538233 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.538249 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.538369 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.538690 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.538693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.539147 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:30.539255 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.539278 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.540377 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.540380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.541127 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.541138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.541719 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.541722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.541832 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.542473 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:30.542485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:30.543904 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:30.545771 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:30.545781 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:30.545786 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:30.546015 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:30.546303 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:30.546305 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:30.546307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:30.546394 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:30.546807 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:30.546812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:30.546927 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.547016 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.547127 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":32,"thread_start_us":105,"threads_started":1}
I20260923 14:45:30.547449 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.538042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.30.2.254:35891)
0923 14:45:30.538192 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.538198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.538327 (+ 129us) server_negotiation.cc:407] Connection header received
0923 14:45:30.538487 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.538490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.538543 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.538650 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:30.539402 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.540266 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.541237 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.541583 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.541704 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.544040 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:30.544045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:30.544055 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:30.544083 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:30.545662 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:30.546111 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:30.546116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:30.546118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:30.546211 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:30.546479 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:30.546483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:30.546485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:30.546700 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:30.546817 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.547153 (+ 336us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.547275 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":50}
I20260923 14:45:30.551265 31853 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:51276:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 14:45:30.552229 31853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:30.557281 32133 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf00de7048142dbafa1c1bd99f38840 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83189bf51ab8499193d932e90d8c7614]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:45:30.557281 31993 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf00de7048142dbafa1c1bd99f38840 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83189bf51ab8499193d932e90d8c7614]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:45:30.557682 31993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf00de7048142dbafa1c1bd99f38840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:30.557682 32133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf00de7048142dbafa1c1bd99f38840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:30.557590 32275 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf00de7048142dbafa1c1bd99f38840 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83189bf51ab8499193d932e90d8c7614]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:45:30.557966 32275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf00de7048142dbafa1c1bd99f38840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:30.560523 32365 tablet_bootstrap.cc:492] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: Bootstrap starting.
I20260923 14:45:30.560837 32364 tablet_bootstrap.cc:492] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: Bootstrap starting.
I20260923 14:45:30.561429 32365 tablet_bootstrap.cc:654] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:30.561466 32366 tablet_bootstrap.cc:492] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82: Bootstrap starting.
I20260923 14:45:30.561874 32364 tablet_bootstrap.cc:654] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:30.562474 32366 tablet_bootstrap.cc:654] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:30.563428 32364 tablet_bootstrap.cc:492] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: No bootstrap required, opened a new log
I20260923 14:45:30.563428 32365 tablet_bootstrap.cc:492] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: No bootstrap required, opened a new log
I20260923 14:45:30.563508 32365 ts_tablet_manager.cc:1397] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 14:45:30.563508 32364 ts_tablet_manager.cc:1397] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260923 14:45:30.563951 32364 raft_consensus.cc:348] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.563978 32365 raft_consensus.cc:348] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.564065 32365 raft_consensus.cc:374] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:30.564062 32364 raft_consensus.cc:374] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:30.564096 32365 raft_consensus.cc:729] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a076f8d22b5e4d59b34e963542dd1b20, State: Initialized, Role: FOLLOWER
I20260923 14:45:30.564110 32364 raft_consensus.cc:729] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ba5786c8204f4aa3cfdd7457a711fc, State: Initialized, Role: FOLLOWER
I20260923 14:45:30.564239 32366 tablet_bootstrap.cc:492] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82: No bootstrap required, opened a new log
I20260923 14:45:30.564221 32364 consensus_queue.cc:260] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.564221 32365 consensus_queue.cc:260] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.564318 32366 ts_tablet_manager.cc:1397] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 14:45:30.564491 32364 ts_tablet_manager.cc:1428] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:30.564524 32365 ts_tablet_manager.cc:1428] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:30.564738 32372 tablet_replica.cc:439] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } } }
I20260923 14:45:30.564785 32371 tablet_replica.cc:439] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } } }
I20260923 14:45:30.564869 32372 tablet_replica.cc:442] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:30.564895 32371 tablet_replica.cc:442] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:30.565121 32366 raft_consensus.cc:348] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.565233 32366 raft_consensus.cc:374] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:30.565279 32366 raft_consensus.cc:729] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a4fc3c45004ef0970ca8f332753c82, State: Initialized, Role: FOLLOWER
I20260923 14:45:30.565407 32366 consensus_queue.cc:260] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.565708 32366 ts_tablet_manager.cc:1428] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 14:45:30.566186 32370 tablet_replica.cc:439] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } } }
I20260923 14:45:30.566334 32370 tablet_replica.cc:442] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 14:45:30.705917 32200 tablet.cc:2369] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:45:30.715667 32060 tablet.cc:2369] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:45:30.861862 32372 raft_consensus.cc:482] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:30.861997 32372 raft_consensus.cc:504] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.862360 32372 leader_election.cc:288] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44ba5786c8204f4aa3cfdd7457a711fc (127.30.2.193:39789), 42a4fc3c45004ef0970ca8f332753c82 (127.30.2.195:44945)
I20260923 14:45:30.862902 32295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf00de7048142dbafa1c1bd99f38840" candidate_uuid: "a076f8d22b5e4d59b34e963542dd1b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a4fc3c45004ef0970ca8f332753c82" is_pre_election: true
I20260923 14:45:30.862902 32013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf00de7048142dbafa1c1bd99f38840" candidate_uuid: "a076f8d22b5e4d59b34e963542dd1b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" is_pre_election: true
I20260923 14:45:30.863063 32295 raft_consensus.cc:2454] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a076f8d22b5e4d59b34e963542dd1b20 in term 0.
I20260923 14:45:30.863063 32013 raft_consensus.cc:2454] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a076f8d22b5e4d59b34e963542dd1b20 in term 0.
I20260923 14:45:30.863386 32087 leader_election.cc:302] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 42a4fc3c45004ef0970ca8f332753c82, 44ba5786c8204f4aa3cfdd7457a711fc, a076f8d22b5e4d59b34e963542dd1b20; no voters:
I20260923 14:45:30.863572 32372 raft_consensus.cc:2781] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:30.863637 32372 raft_consensus.cc:482] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:30.863665 32372 raft_consensus.cc:3037] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:30.864741 32372 raft_consensus.cc:504] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.865080 32372 leader_election.cc:288] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [CANDIDATE]: Term 1 election: Requested vote from peers 44ba5786c8204f4aa3cfdd7457a711fc (127.30.2.193:39789), 42a4fc3c45004ef0970ca8f332753c82 (127.30.2.195:44945)
I20260923 14:45:30.865453 32295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf00de7048142dbafa1c1bd99f38840" candidate_uuid: "a076f8d22b5e4d59b34e963542dd1b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a4fc3c45004ef0970ca8f332753c82"
I20260923 14:45:30.865463 32013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf00de7048142dbafa1c1bd99f38840" candidate_uuid: "a076f8d22b5e4d59b34e963542dd1b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ba5786c8204f4aa3cfdd7457a711fc"
I20260923 14:45:30.865585 32295 raft_consensus.cc:3037] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:30.865597 32013 raft_consensus.cc:3037] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:30.866473 32013 raft_consensus.cc:2454] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a076f8d22b5e4d59b34e963542dd1b20 in term 1.
I20260923 14:45:30.866557 32295 raft_consensus.cc:2454] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a076f8d22b5e4d59b34e963542dd1b20 in term 1.
I20260923 14:45:30.866773 32087 leader_election.cc:302] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ba5786c8204f4aa3cfdd7457a711fc, a076f8d22b5e4d59b34e963542dd1b20; no voters:
I20260923 14:45:30.867017 32372 raft_consensus.cc:2781] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:30.867117 32372 raft_consensus.cc:686] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [term 1 LEADER]: Becoming Leader. State: Replica: a076f8d22b5e4d59b34e963542dd1b20, State: Running, Role: LEADER
I20260923 14:45:30.867269 32372 consensus_queue.cc:237] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } }
I20260923 14:45:30.867981 32374 tablet_replica.cc:439] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: TxnStatusTablet state changed. Reason: New leader a076f8d22b5e4d59b34e963542dd1b20. Latest consensus state: current_term: 1 leader_uuid: "a076f8d22b5e4d59b34e963542dd1b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } } }
I20260923 14:45:30.868103 32374 tablet_replica.cc:442] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:45:30.868335 32405 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 14:45:30.868503 32405 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 14:45:30.869076 31853 catalog_manager.cc:5495] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to a076f8d22b5e4d59b34e963542dd1b20 (127.30.2.194). New cstate: current_term: 1 leader_uuid: "a076f8d22b5e4d59b34e963542dd1b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } health_report { overall_health: UNKNOWN } } }
I20260923 14:45:30.896109 32370 consensus_queue.cc:1035] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:30.900357 32408 consensus_queue.cc:1035] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:30.933609 32400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.930232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41983 (local address 127.0.0.1:33468)
0923 14:45:30.930373 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.930390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.930504 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.930811 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.930814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.930822 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:45:30.930924 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.930930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.931644 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.931648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.932417 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.932427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.932898 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.932901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.932992 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.933035 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:45:30.933326 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:45:30.933344 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.933409 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.933485 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 14:45:30.933606 32359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.930342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.30.2.194:41983)
0923 14:45:30.930485 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.930490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.930505 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:45:30.930651 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.930656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.930708 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.930786 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:30.931026 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.931529 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.932539 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.932787 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.932896 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.933178 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:30.933245 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:30.933319 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.933419 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.933466 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 14:45:30.936192 32295 raft_consensus.cc:1261] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82 [term 1 FOLLOWER]: Refusing update from remote peer a076f8d22b5e4d59b34e963542dd1b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:30.936255 32013 raft_consensus.cc:1261] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc [term 1 FOLLOWER]: Refusing update from remote peer a076f8d22b5e4d59b34e963542dd1b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:30.936587 32374 consensus_queue.cc:1035] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:30.936916 32374 consensus_queue.cc:1035] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:30.937953 32408 tablet_replica.cc:439] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82: TxnStatusTablet state changed. Reason: New leader a076f8d22b5e4d59b34e963542dd1b20. Latest consensus state: current_term: 1 leader_uuid: "a076f8d22b5e4d59b34e963542dd1b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } } }
I20260923 14:45:30.938062 32408 tablet_replica.cc:442] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:30.938265 32371 tablet_replica.cc:439] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: TxnStatusTablet state changed. Reason: New leader a076f8d22b5e4d59b34e963542dd1b20. Latest consensus state: current_term: 1 leader_uuid: "a076f8d22b5e4d59b34e963542dd1b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } } }
I20260923 14:45:30.938371 32371 tablet_replica.cc:442] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:30.939397 32374 tablet_replica.cc:439] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a076f8d22b5e4d59b34e963542dd1b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } } }
I20260923 14:45:30.939518 32374 tablet_replica.cc:442] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:45:30.939653 32377 mvcc.cc:205] Tried to move back new op lower bound from 7332555697910775808 to 7332555697633656832. Current Snapshot: MvccSnapshot[applied={T|T < 7332555697910775808}]
I20260923 14:45:30.941723 32408 tablet_replica.cc:439] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a076f8d22b5e4d59b34e963542dd1b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } } }
I20260923 14:45:30.941821 32408 tablet_replica.cc:442] T 0cf00de7048142dbafa1c1bd99f38840 P 42a4fc3c45004ef0970ca8f332753c82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:30.942574 32374 tablet_replica.cc:439] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a076f8d22b5e4d59b34e963542dd1b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } } }
I20260923 14:45:30.942687 32374 tablet_replica.cc:442] T 0cf00de7048142dbafa1c1bd99f38840 P a076f8d22b5e4d59b34e963542dd1b20: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:45:30.944172 32371 tablet_replica.cc:439] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a076f8d22b5e4d59b34e963542dd1b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ba5786c8204f4aa3cfdd7457a711fc" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39789 } } peers { permanent_uuid: "a076f8d22b5e4d59b34e963542dd1b20" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41983 } } peers { permanent_uuid: "42a4fc3c45004ef0970ca8f332753c82" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 44945 } } }
I20260923 14:45:30.944286 32371 tablet_replica.cc:442] T 0cf00de7048142dbafa1c1bd99f38840 P 44ba5786c8204f4aa3cfdd7457a711fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:30.951263 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.946680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.30.2.195:44945)
0923 14:45:30.947223 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.947228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.947240 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:45:30.947316 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.947319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.947379 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.947498 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:30.947734 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.948651 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.949530 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.949930 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.950038 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.950308 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:30.950870 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:30.950985 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.951053 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.951108 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":51,"threads_started":1}
I20260923 14:45:30.965201 32359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.960885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:57017 (local address 127.30.2.194:41983)
0923 14:45:30.960994 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.960999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.961175 (+ 176us) server_negotiation.cc:407] Connection header received
0923 14:45:30.961303 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.961307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.961362 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.961449 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:30.961715 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.962534 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.963935 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.964297 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.964409 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.964738 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:30.964816 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:30.964892 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.965014 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.965072 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:45:30.965230 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.960736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41983 (local address 127.30.2.195:57017)
0923 14:45:30.961081 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.961096 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.961200 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.961475 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.961478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.961486 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:45:30.961600 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.961606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.962675 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.962679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.963743 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.963755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.964425 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.964430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.964540 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.964591 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:45:30.964910 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:45:30.964917 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.964990 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.965071 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":100,"threads_started":1}
I20260923 14:45:30.995342 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.991383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:59025 (local address 127.30.2.195:44945)
0923 14:45:30.991501 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:30.991507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:30.991680 (+ 173us) server_negotiation.cc:407] Connection header received
0923 14:45:30.991811 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:30.991815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:30.991874 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:30.991966 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:30.992243 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.993112 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.994200 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.994445 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.994579 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.994881 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:30.994965 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:30.995049 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:30.995153 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 14:45:30.995204 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:45:30.995368 32421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:30.991293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:44945 (local address 127.30.2.194:59025)
0923 14:45:30.991588 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:30.991603 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:30.991698 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:30.991983 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:30.991985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:30.991992 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:45:30.992125 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.992130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.993215 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.993219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.994075 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:30.994086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:30.994583 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:30.994587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:30.994684 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:30.994737 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:45:30.995064 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:45:30.995073 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 14:45:30.995144 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 14:45:30.995229 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260923 14:45:31.022881 31854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:31.023085 31854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:31.025522 31854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 904a83df8b024cea86adcde46992bafc: Sending DeleteTablet for 3 replicas of tablet ff377f25eb33468a9543f3d9a8d312fe
I20260923 14:45:31.026526 31993 tablet_service.cc:1514] Processing DeleteTablet for tablet ff377f25eb33468a9543f3d9a8d312fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:31 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:54494
I20260923 14:45:31.027129 32275 tablet_service.cc:1514] Processing DeleteTablet for tablet ff377f25eb33468a9543f3d9a8d312fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:31 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:54936
I20260923 14:45:31.027489 32133 tablet_service.cc:1514] Processing DeleteTablet for tablet ff377f25eb33468a9543f3d9a8d312fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:31 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:33450
I20260923 14:45:31.027953 32424 tablet_replica.cc:330] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc: stopping tablet replica
I20260923 14:45:31.028134 32424 raft_consensus.cc:2229] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:31.028177 32426 tablet_replica.cc:330] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82: stopping tablet replica
I20260923 14:45:31.028294 32424 raft_consensus.cc:2258] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:31.028311 32426 raft_consensus.cc:2229] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:45:31.028563 32428 tablet_replica.cc:330] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20: stopping tablet replica
I20260923 14:45:31.028628 32426 raft_consensus.cc:2258] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:31.028698 32428 raft_consensus.cc:2229] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:31.028824 32428 raft_consensus.cc:2258] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:31.029378 32424 ts_tablet_manager.cc:1905] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:31.029381 32426 ts_tablet_manager.cc:1905] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:31.032411 32424 ts_tablet_manager.cc:1918] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:45:31.032469 32426 ts_tablet_manager.cc:1918] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:45:31.032531 32424 log.cc:1196] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/ff377f25eb33468a9543f3d9a8d312fe
I20260923 14:45:31.032564 32426 log.cc:1196] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/ff377f25eb33468a9543f3d9a8d312fe
I20260923 14:45:31.032822 32424 ts_tablet_manager.cc:1939] T ff377f25eb33468a9543f3d9a8d312fe P 44ba5786c8204f4aa3cfdd7457a711fc: Deleting consensus metadata
I20260923 14:45:31.032876 32426 ts_tablet_manager.cc:1939] T ff377f25eb33468a9543f3d9a8d312fe P 42a4fc3c45004ef0970ca8f332753c82: Deleting consensus metadata
I20260923 14:45:31.033524 31842 catalog_manager.cc:4841] TS 44ba5786c8204f4aa3cfdd7457a711fc (127.30.2.193:39789): tablet ff377f25eb33468a9543f3d9a8d312fe (table test-table [id=05cca5002ddb4c78bb3a4a5266e3bbab]) successfully deleted
I20260923 14:45:31.033625 31840 catalog_manager.cc:4841] TS 42a4fc3c45004ef0970ca8f332753c82 (127.30.2.195:44945): tablet ff377f25eb33468a9543f3d9a8d312fe (table test-table [id=05cca5002ddb4c78bb3a4a5266e3bbab]) successfully deleted
I20260923 14:45:31.034297 32428 ts_tablet_manager.cc:1905] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:31.037309 32428 ts_tablet_manager.cc:1918] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:45:31.037395 32428 log.cc:1196] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/ff377f25eb33468a9543f3d9a8d312fe
I20260923 14:45:31.037681 32428 ts_tablet_manager.cc:1939] T ff377f25eb33468a9543f3d9a8d312fe P a076f8d22b5e4d59b34e963542dd1b20: Deleting consensus metadata
I20260923 14:45:31.038393 31842 catalog_manager.cc:4841] TS a076f8d22b5e4d59b34e963542dd1b20 (127.30.2.194:41983): tablet ff377f25eb33468a9543f3d9a8d312fe (table test-table [id=05cca5002ddb4c78bb3a4a5266e3bbab]) successfully deleted
Sep 23 14:45:31 dist-test-slave-wjf2 krb5kdc[31810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174730, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:45:31.048648 32427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:31.027513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.30.2.193:39789)
0923 14:45:31.033920 (+ 6407us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:31.033926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:31.036123 (+ 2197us) server_negotiation.cc:407] Connection header received
0923 14:45:31.036314 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:31.036318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:31.036381 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:31.036478 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:31.037626 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:31.038560 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:31.040106 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:31.040467 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:31.040572 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:31.044577 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:31.044589 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:31.044608 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:31.044642 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:31.046506 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:31.047020 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:31.047027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:31.047028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:31.047109 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:31.047418 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:31.047421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:31.047423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:31.047694 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:31.047961 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:31.048300 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 14:45:31.048432 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6318,"thread_start_us":74,"threads_started":1}
W20260923 14:45:31.049098 32053 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54502
I20260923 14:45:31.059355 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:31.050909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51284 (local address 127.30.2.254:35891)
0923 14:45:31.051167 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:31.051171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:31.051185 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:45:31.051287 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:31.051289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:31.051329 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:31.051404 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:31.052315 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:31.052854 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:31.054091 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:31.054317 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:31.054402 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:31.055947 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:31.055950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:31.055952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:31.055976 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:31.057623 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:31.058054 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:31.058057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:31.058058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:31.058164 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:31.058482 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:31.058485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:31.058486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:31.058674 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:31.058787 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:31.059037 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 14:45:31.059146 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":56,"threads_started":1}
W20260923 14:45:31.059770 31854 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51284
I20260923 14:45:31.061146 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 31928
I20260923 14:45:31.069610 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 32068
I20260923 14:45:31.077675 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 32208
I20260923 14:45:31.082844 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 31823
2026-09-23T14:45:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3019 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:31 dist-test-slave-wjf2 krb5kdc[32435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:31 dist-test-slave-wjf2 krb5kdc[32435](info): set up 2 sockets
Sep 23 14:45:31 dist-test-slave-wjf2 krb5kdc[32435](info): commencing operation
krb5kdc: starting...
W20260923 14:45:32.128971 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:32 dist-test-slave-wjf2 krb5kdc[32435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:45:32.246584 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:41293
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:32.354008 32448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:32.354249 32448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:32.354286 32448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:32.358270 32448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:32.358319 32448 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 14:45:32.358336 32448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:32.358353 32448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:32.358388 32448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:32.358436 32448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:32.363471 32448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:41293
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:41293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:32.364573 32448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:32.365609 32448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:32.374387 32454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:32.374392 32456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:32.374487 32448 server_base.cc:1034] running on GCE node
W20260923 14:45:32.374392 32453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:32.375108 32448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:32.376334 32448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:32.377539 32448 hybrid_clock.cc:648] HybridClock initialized: now 1790174732377520 us; error 31 us; skew 500 ppm
Sep 23 14:45:32 dist-test-slave-wjf2 krb5kdc[32435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174732, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:32.380463 32448 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:45:32.381438 32448 webserver.cc:460] Webserver started at http://127.30.2.254:38331/ using document root <none> and password file <none>
I20260923 14:45:32.382023 32448 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:32.382122 32448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:32.382364 32448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:32.384606 32448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "d3f92d71cfc84cf9aeea31bf0090f150"
format_stamp: "Formatted at 2026-09-23 14:45:32 on dist-test-slave-wjf2"
server_key: "bff4ac85fd41c60caf3a6e6c48031316"
server_key_iv: "b031836cda6e4ea043651bccedff83eb"
server_key_version: "encryptionkey@0"
I20260923 14:45:32.385205 32448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "d3f92d71cfc84cf9aeea31bf0090f150"
format_stamp: "Formatted at 2026-09-23 14:45:32 on dist-test-slave-wjf2"
server_key: "bff4ac85fd41c60caf3a6e6c48031316"
server_key_iv: "b031836cda6e4ea043651bccedff83eb"
server_key_version: "encryptionkey@0"
I20260923 14:45:32.389230 32448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:45:32.391710 32463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:32.393034 32448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:32.393186 32448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "d3f92d71cfc84cf9aeea31bf0090f150"
format_stamp: "Formatted at 2026-09-23 14:45:32 on dist-test-slave-wjf2"
server_key: "bff4ac85fd41c60caf3a6e6c48031316"
server_key_iv: "b031836cda6e4ea043651bccedff83eb"
server_key_version: "encryptionkey@0"
I20260923 14:45:32.393302 32448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:32.403962 32448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:32.406941 32448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:32.407143 32448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:32.415093 32448 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:41293
I20260923 14:45:32.415169 32525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:41293 every 8 connection(s)
I20260923 14:45:32.416076 32448 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:32.419029 32526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:32.422215 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 32448
I20260923 14:45:32.422314 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:32.422603 30731 external_mini_cluster.cc:1428] Setting key 95de86afd76bec26851044466229393c
I20260923 14:45:32.425673 32526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150: Bootstrap starting.
I20260923 14:45:32.428755 32526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:32.429639 32526 log.cc:824] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:32.432159 32526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150: No bootstrap required, opened a new log
Sep 23 14:45:32 dist-test-slave-wjf2 krb5kdc[32435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:32.437278 32526 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f92d71cfc84cf9aeea31bf0090f150" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41293 } }
I20260923 14:45:32.437615 32526 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:32.437717 32526 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f92d71cfc84cf9aeea31bf0090f150, State: Initialized, Role: FOLLOWER
I20260923 14:45:32.438290 32526 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f92d71cfc84cf9aeea31bf0090f150" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41293 } }
I20260923 14:45:32.438457 32526 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:32.438536 32526 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:32.438639 32526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:32.439411 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:32.424148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.30.2.254:41293)
0923 14:45:32.424621 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:32.424642 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:32.424688 (+ 46us) server_negotiation.cc:407] Connection header received
0923 14:45:32.425538 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:32.425582 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:32.426086 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:32.426490 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:32.427266 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.429011 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.430056 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.430721 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.430885 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:32.433627 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:32.433634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:32.433652 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:32.433687 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:32.435927 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:32.436380 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:32.436387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:32.436394 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:32.436549 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:32.436838 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:32.436841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:32.436843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:32.437403 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:32.437642 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:32.437988 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 14:45:32.438316 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260923 14:45:32.439942 32526 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f92d71cfc84cf9aeea31bf0090f150" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41293 } }
I20260923 14:45:32.440335 32526 leader_election.cc:302] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3f92d71cfc84cf9aeea31bf0090f150; no voters:
I20260923 14:45:32.440698 32526 leader_election.cc:288] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:32.440819 32531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:32.441071 32531 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [term 1 LEADER]: Becoming Leader. State: Replica: d3f92d71cfc84cf9aeea31bf0090f150, State: Running, Role: LEADER
I20260923 14:45:32.441505 32531 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f92d71cfc84cf9aeea31bf0090f150" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41293 } }
I20260923 14:45:32.442279 32526 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:32.442900 32531 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3f92d71cfc84cf9aeea31bf0090f150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f92d71cfc84cf9aeea31bf0090f150" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41293 } } }
I20260923 14:45:32.443138 32531 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:32.443377 32532 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3f92d71cfc84cf9aeea31bf0090f150. Latest consensus state: current_term: 1 leader_uuid: "d3f92d71cfc84cf9aeea31bf0090f150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f92d71cfc84cf9aeea31bf0090f150" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41293 } } }
I20260923 14:45:32.443563 32532 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:32.443629 32538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:32.447263 32538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:32.453933 32538 catalog_manager.cc:1347] Generated new cluster ID: c775ce04fc654c73b3292fec11107933
I20260923 14:45:32.454030 32538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:32.480369 32538 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:32.481482 32538 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:32.485489 32538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150: Generated new TSK 0
I20260923 14:45:32.486325 32538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:45:32.521404 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:41293
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:32.637890 32552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:32.638165 32552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:32.638245 32552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:32.642342 32552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:32.642416 32552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:32.642478 32552 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 14:45:32.642516 32552 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:45:32.642588 32552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:32.648141 32552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.2.254:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:32.649199 32552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:32.650251 32552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:32.659727 32558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:32.659763 32557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:32.659902 32560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:32.660133 32552 server_base.cc:1034] running on GCE node
I20260923 14:45:32.660507 32552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:32.661146 32552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:32.662365 32552 hybrid_clock.cc:648] HybridClock initialized: now 1790174732662346 us; error 30 us; skew 500 ppm
Sep 23 14:45:32 dist-test-slave-wjf2 krb5kdc[32435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174732, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:32.665306 32552 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:45:32.666244 32552 webserver.cc:460] Webserver started at http://127.30.2.193:33253/ using document root <none> and password file <none>
I20260923 14:45:32.666853 32552 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:32.666956 32552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:32.667204 32552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:32.669410 32552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "38cef62d399340848985c5cc600c03f9"
format_stamp: "Formatted at 2026-09-23 14:45:32 on dist-test-slave-wjf2"
server_key: "f3036386075bf503623b6d57d36a816d"
server_key_iv: "515acec7e5a1dbaebadeebeaa25e568f"
server_key_version: "encryptionkey@0"
I20260923 14:45:32.669996 32552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "38cef62d399340848985c5cc600c03f9"
format_stamp: "Formatted at 2026-09-23 14:45:32 on dist-test-slave-wjf2"
server_key: "f3036386075bf503623b6d57d36a816d"
server_key_iv: "515acec7e5a1dbaebadeebeaa25e568f"
server_key_version: "encryptionkey@0"
I20260923 14:45:32.673807 32552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:45:32.676311 32567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:32.677695 32552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:32.677850 32552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "38cef62d399340848985c5cc600c03f9"
format_stamp: "Formatted at 2026-09-23 14:45:32 on dist-test-slave-wjf2"
server_key: "f3036386075bf503623b6d57d36a816d"
server_key_iv: "515acec7e5a1dbaebadeebeaa25e568f"
server_key_version: "encryptionkey@0"
I20260923 14:45:32.677978 32552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:32.692636 32552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:32.695771 32552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:32.696043 32552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:32.697716 32552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:32.697790 32552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:32.697863 32552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:32.697912 32552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:45:32 dist-test-slave-wjf2 krb5kdc[32435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174732, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:32.710570 32552 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:40767
I20260923 14:45:32.710702 32682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:40767 every 8 connection(s)
I20260923 14:45:32.711807 32552 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:45:32.713076 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:32.698596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:41785 (local address 127.30.2.254:41293)
0923 14:45:32.698716 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:32.698719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:32.699903 (+ 1184us) server_negotiation.cc:407] Connection header received
0923 14:45:32.700900 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:32.700903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:32.700964 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:32.701045 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:32.702290 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.703018 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.704272 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.704800 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.704947 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:32.708936 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:32.708939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:32.708941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:32.708969 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:32.710719 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:32.711349 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:32.711352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:32.711353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:32.711430 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:32.711848 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:32.711852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:32.711854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:32.712061 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:32.712138 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:32.712848 (+ 710us) server_negotiation.cc:299] Negotiation successful
0923 14:45:32.712955 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:45:32.714051 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:32.699221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41293 (local address 127.30.2.193:41785)
0923 14:45:32.699723 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:32.699765 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:32.700696 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:32.701256 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:32.701267 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:32.701670 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:32.702125 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:32.702138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.703141 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.703145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:32.704084 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:32.704110 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.704974 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.704978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:32.705208 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:32.705963 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:32.705979 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:32.708707 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:32.710848 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:32.710854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:32.710869 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:32.711233 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:32.711524 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:32.711528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:32.711530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:32.711715 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:32.712179 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:32.712187 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:32.712533 (+ 346us) client_negotiation.cc:769] Sending connection context
0923 14:45:32.712793 (+ 260us) client_negotiation.cc:240] Negotiation successful
0923 14:45:32.713126 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":151,"threads_started":1}
I20260923 14:45:32.715372 32683 heartbeater.cc:347] Connected to a master server at 127.30.2.254:41293
I20260923 14:45:32.715538 32683 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:32.716130 32683 heartbeater.cc:511] Master 127.30.2.254:41293 requested a full tablet report, sending...
I20260923 14:45:32.717839 32479 ts_manager.cc:194] Registered new tserver with Master: 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767)
I20260923 14:45:32.718400 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 32552
I20260923 14:45:32.718529 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:32.718815 30731 external_mini_cluster.cc:1428] Setting key d92949ac2d71df294811477df940ab47
I20260923 14:45:32.719580 32479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:41785
I20260923 14:45:32.725745 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:32.717441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41293 (local address 127.30.2.193:39849)
0923 14:45:32.717762 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:32.717778 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:32.717889 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:32.718149 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:32.718152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:32.718369 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:32.718495 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:32.718502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.719716 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.719719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:32.720539 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:32.720550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.721054 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.721058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:32.721235 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:32.721805 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:32.721827 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:32.722348 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:32.724109 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:32.724117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:32.724122 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:32.724420 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:32.724734 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:32.724740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:32.724744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:32.724861 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:32.725242 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:32.725248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:32.725370 (+ 122us) client_negotiation.cc:769] Sending connection context
0923 14:45:32.725477 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 14:45:32.725585 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":33,"thread_start_us":87,"threads_started":1}
I20260923 14:45:32.725968 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:32.717566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:39849 (local address 127.30.2.254:41293)
0923 14:45:32.717689 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:32.717693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:32.717851 (+ 158us) server_negotiation.cc:407] Connection header received
0923 14:45:32.717983 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:32.717986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:32.718031 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:32.718103 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:32.718664 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.719514 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.720677 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.720930 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.721049 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:32.722488 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:32.722491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:32.722493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:32.722514 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:32.723996 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:32.724532 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:32.724537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:32.724540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:32.724624 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:32.724958 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:32.724960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:32.724961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:32.725114 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:32.725217 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:32.725616 (+ 399us) server_negotiation.cc:299] Negotiation successful
0923 14:45:32.725748 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:45:32.746661 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:41293
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:32.863140 32692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:32.863412 32692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:32.863464 32692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:32.868100 32692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:32.868158 32692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:32.868202 32692 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 14:45:32.868218 32692 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:45:32.868254 32692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:32.873929 32692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.2.254:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:32.874946 32692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:32.876049 32692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:32.886399 32698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:32.886448 32700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:32.886507 32692 server_base.cc:1034] running on GCE node
W20260923 14:45:32.886451 32697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:32.887163 32692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:32.887893 32692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:32.889058 32692 hybrid_clock.cc:648] HybridClock initialized: now 1790174732889015 us; error 51 us; skew 500 ppm
Sep 23 14:45:32 dist-test-slave-wjf2 krb5kdc[32435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174732, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:32.891966 32692 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:45:32.892926 32692 webserver.cc:460] Webserver started at http://127.30.2.194:41485/ using document root <none> and password file <none>
I20260923 14:45:32.893523 32692 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:32.893589 32692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:32.893826 32692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:32.896005 32692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "6be15d44699b45ba8e6dcf690a1280bf"
format_stamp: "Formatted at 2026-09-23 14:45:32 on dist-test-slave-wjf2"
server_key: "6dd383b6f75344725a5a56d0903c78c8"
server_key_iv: "ff5a90b0793db16ed60a65ad449f4d32"
server_key_version: "encryptionkey@0"
I20260923 14:45:32.896591 32692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "6be15d44699b45ba8e6dcf690a1280bf"
format_stamp: "Formatted at 2026-09-23 14:45:32 on dist-test-slave-wjf2"
server_key: "6dd383b6f75344725a5a56d0903c78c8"
server_key_iv: "ff5a90b0793db16ed60a65ad449f4d32"
server_key_version: "encryptionkey@0"
I20260923 14:45:32.900538 32692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 14:45:32.903047 32707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:32.904326 32692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260923 14:45:32.904480 32692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "6be15d44699b45ba8e6dcf690a1280bf"
format_stamp: "Formatted at 2026-09-23 14:45:32 on dist-test-slave-wjf2"
server_key: "6dd383b6f75344725a5a56d0903c78c8"
server_key_iv: "ff5a90b0793db16ed60a65ad449f4d32"
server_key_version: "encryptionkey@0"
I20260923 14:45:32.904590 32692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:32.915845 32692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:32.918812 32692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:32.919001 32692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:32.920512 32692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:32.920578 32692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:32.920662 32692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:32.920712 32692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:45:32 dist-test-slave-wjf2 krb5kdc[32435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174732, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:32.931993 32692 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:41029
I20260923 14:45:32.932184 354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:41029 every 8 connection(s)
I20260923 14:45:32.933142 32692 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:32.935170 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:32.921266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:46819 (local address 127.30.2.254:41293)
0923 14:45:32.921395 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:32.921399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:32.923020 (+ 1621us) server_negotiation.cc:407] Connection header received
0923 14:45:32.924341 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:32.924344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:32.924409 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:32.924511 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:32.925815 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.926518 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.927976 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.928226 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.928346 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:32.931059 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:32.931063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:32.931066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:32.931096 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:32.932988 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:32.933620 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:32.933624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:32.933626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:32.933733 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:32.934165 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:32.934168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:32.934170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:32.934366 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:32.934449 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:32.934904 (+ 455us) server_negotiation.cc:299] Negotiation successful
0923 14:45:32.935006 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:45:32.935976 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:32.922286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41293 (local address 127.30.2.194:46819)
0923 14:45:32.922849 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:32.922907 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:32.924116 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:32.924731 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:32.924741 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:32.925195 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:32.925665 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:32.925679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.926648 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.926654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:32.927853 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:32.927863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.928395 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.928400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:32.928577 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:32.929174 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:32.929191 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:32.930886 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:32.933135 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:32.933140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:32.933151 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:32.933484 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:32.933849 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:32.933852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:32.933854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:32.934030 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:32.934460 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:32.934465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:32.934686 (+ 221us) client_negotiation.cc:769] Sending connection context
0923 14:45:32.934845 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 14:45:32.935081 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":137,"threads_started":1}
I20260923 14:45:32.937081 355 heartbeater.cc:347] Connected to a master server at 127.30.2.254:41293
I20260923 14:45:32.937218 355 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:32.937814 355 heartbeater.cc:511] Master 127.30.2.254:41293 requested a full tablet report, sending...
I20260923 14:45:32.938978 32480 ts_manager.cc:194] Registered new tserver with Master: 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194:41029)
I20260923 14:45:32.939855 32480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:46819
I20260923 14:45:32.943357 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 32692
I20260923 14:45:32.943482 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:32.943866 30731 external_mini_cluster.cc:1428] Setting key 47f9a99cdd796e5870707cfaba1652e2
I20260923 14:45:32.947351 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:32.939469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41293 (local address 127.30.2.194:58007)
0923 14:45:32.939846 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:32.939860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:32.939958 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:32.940276 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:32.940279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:32.940486 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:32.940596 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:32.940602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.941701 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.941704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:32.942251 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:32.942259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.942742 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.942745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:32.942813 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:32.943406 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:32.943418 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:32.943858 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:32.945695 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:32.945700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:32.945702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:32.945900 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:32.946206 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:32.946209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:32.946210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:32.946288 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:32.946916 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:32.946919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:32.946996 (+ 77us) client_negotiation.cc:769] Sending connection context
0923 14:45:32.947087 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 14:45:32.947211 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":87,"threads_started":1}
I20260923 14:45:32.947508 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:32.939537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:58007 (local address 127.30.2.254:41293)
0923 14:45:32.939660 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:32.939663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:32.939926 (+ 263us) server_negotiation.cc:407] Connection header received
0923 14:45:32.940086 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:32.940090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:32.940139 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:32.940239 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:32.940715 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.941548 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.942354 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:32.942575 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:32.942727 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:32.944024 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:32.944029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:32.944032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:32.944060 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:32.945571 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:32.946003 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:32.946008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:32.946010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:32.946098 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:32.946384 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:32.946389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:32.946391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:32.946792 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:32.946928 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:32.947171 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 14:45:32.947302 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:45:32.970860 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:41293
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:33.082335 364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:33.082597 364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:33.082696 364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:33.086761 364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:33.086838 364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:33.086913 364 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 14:45:33.086951 364 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:45:33.087025 364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:33.092337 364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.2.254:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:33.093413 364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:33.094437 364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:33.104382 372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:33.104391 370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:33.104445 364 server_base.cc:1034] running on GCE node
W20260923 14:45:33.104391 369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:33.104935 364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:33.105536 364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:33.106709 364 hybrid_clock.cc:648] HybridClock initialized: now 1790174733106675 us; error 44 us; skew 500 ppm
Sep 23 14:45:33 dist-test-slave-wjf2 krb5kdc[32435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174733, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:33.109709 364 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:45:33.110512 364 webserver.cc:460] Webserver started at http://127.30.2.195:34415/ using document root <none> and password file <none>
I20260923 14:45:33.111049 364 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:33.111107 364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:33.111334 364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:33.113306 364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "8dae3adade4a44a6b5efae6081091956"
format_stamp: "Formatted at 2026-09-23 14:45:33 on dist-test-slave-wjf2"
server_key: "0f33bf772a8034a81b97b70f3aeee9d3"
server_key_iv: "9b990a904e870b8c137133bea2229291"
server_key_version: "encryptionkey@0"
I20260923 14:45:33.113912 364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "8dae3adade4a44a6b5efae6081091956"
format_stamp: "Formatted at 2026-09-23 14:45:33 on dist-test-slave-wjf2"
server_key: "0f33bf772a8034a81b97b70f3aeee9d3"
server_key_iv: "9b990a904e870b8c137133bea2229291"
server_key_version: "encryptionkey@0"
I20260923 14:45:33.117525 364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260923 14:45:33.119848 379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:33.121024 364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:45:33.121132 364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "8dae3adade4a44a6b5efae6081091956"
format_stamp: "Formatted at 2026-09-23 14:45:33 on dist-test-slave-wjf2"
server_key: "0f33bf772a8034a81b97b70f3aeee9d3"
server_key_iv: "9b990a904e870b8c137133bea2229291"
server_key_version: "encryptionkey@0"
I20260923 14:45:33.121229 364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:33.158421 364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:33.161518 364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:33.161718 364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:33.163148 364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:33.163196 364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:33.163239 364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:33.163308 364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:45:33 dist-test-slave-wjf2 krb5kdc[32435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174733, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:33.173911 364 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:33889
I20260923 14:45:33.174017 494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:33889 every 8 connection(s)
I20260923 14:45:33.174919 364 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:45:33.176787 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 364
I20260923 14:45:33.176916 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:45:33.177214 30731 external_mini_cluster.cc:1428] Setting key 2519955d00aa1e8231bd9d2510c4c3f9
I20260923 14:45:33.177336 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.164189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:33181 (local address 127.30.2.254:41293)
0923 14:45:33.164353 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.164359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.165334 (+ 975us) server_negotiation.cc:407] Connection header received
0923 14:45:33.166704 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.166710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.166772 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.166881 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:33.168514 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.169160 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.170305 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.170526 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.170641 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.173601 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:33.173608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:33.173614 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:33.173648 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:33.175197 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:33.175764 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:33.175767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:33.175769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:33.175887 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:33.176218 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:33.176221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:33.176222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:33.176396 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:33.176473 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.177013 (+ 540us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.177165 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:45:33.178189 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.164649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41293 (local address 127.30.2.195:33181)
0923 14:45:33.165138 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.165209 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.166447 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.167089 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.167098 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.167588 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:33.168367 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.168394 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.169290 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.169293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.170169 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.170179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.170689 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.170694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.170852 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.171986 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:33.172011 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:33.173443 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:33.175313 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:33.175321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:33.175339 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:33.175666 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:33.175987 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:33.175990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:33.175991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:33.176118 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:33.176499 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:33.176504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:33.176754 (+ 250us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.176950 (+ 196us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.177257 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260923 14:45:33.179430 495 heartbeater.cc:347] Connected to a master server at 127.30.2.254:41293
I20260923 14:45:33.179580 495 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:33.180152 495 heartbeater.cc:511] Master 127.30.2.254:41293 requested a full tablet report, sending...
I20260923 14:45:33.181124 32480 ts_manager.cc:194] Registered new tserver with Master: 8dae3adade4a44a6b5efae6081091956 (127.30.2.195:33889)
I20260923 14:45:33.181879 32480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:33181
I20260923 14:45:33.187659 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.181437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41293 (local address 127.30.2.195:42353)
0923 14:45:33.181725 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.181739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.181829 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.182076 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.182079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.182277 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:33.182363 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.182369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.183153 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.183155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.183715 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.183722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.184146 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.184148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.184221 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.184592 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:33.184603 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:33.184883 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:33.186464 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:33.186467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:33.186468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:33.186651 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:33.186915 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:33.186917 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:33.186918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:33.186990 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:33.187281 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:33.187283 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:33.187353 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.187432 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.187525 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260923 14:45:33.187664 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.181549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:42353 (local address 127.30.2.254:41293)
0923 14:45:33.181706 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.181709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.181803 (+ 94us) server_negotiation.cc:407] Connection header received
0923 14:45:33.181933 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.181937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.181980 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.182059 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:33.182486 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.182986 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.183847 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.184057 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.184137 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.184996 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:33.184999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:33.185001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:33.185021 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:33.186364 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:33.186739 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:33.186741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:33.186743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:33.186818 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:33.187062 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:33.187064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:33.187065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:33.187207 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:33.187278 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.187484 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.187564 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 14:45:33.190846 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 14:45:33 dist-test-slave-wjf2 krb5kdc[32435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 14:45:33 dist-test-slave-wjf2 krb5kdc[32435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:33.215694 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.207223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.30.2.254:41293)
0923 14:45:33.207349 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.207353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.207363 (+ 10us) server_negotiation.cc:407] Connection header received
0923 14:45:33.207400 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.207403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.207439 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.207527 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:33.208334 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.208906 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.209752 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.209958 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.210061 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.212442 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:33.212445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:33.212447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:33.212469 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:33.213884 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:33.214269 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:33.214271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:33.214273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:33.214355 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:33.214767 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:33.214770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:33.214772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:33.214945 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:33.215059 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.215473 (+ 414us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.215577 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 14:45:33.218561 32480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:45:33.221606 32480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:33.236040 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.231331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.30.2.195:33889)
0923 14:45:33.231675 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.231687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.231707 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:45:33.231814 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.231819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.231990 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.232379 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:33.232503 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.233556 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.234619 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.235352 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.235473 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.235641 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.235763 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.235886 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":111,"threads_started":1}
I20260923 14:45:33.236029 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.230938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33889 (local address 127.0.0.1:42810)
0923 14:45:33.231330 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.231342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.231415 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.232135 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.232140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.232162 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:33.232243 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.232249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.233662 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.233666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.234475 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.234486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.235473 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.235477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.235577 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.235615 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.235733 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.235893 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260923 14:45:33.236198 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.230618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:40767 (local address 127.0.0.1:45366)
0923 14:45:33.230912 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.230967 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.231097 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.231876 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.231885 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.231914 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:33.232075 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.232083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.233215 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.233221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.234580 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.234592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.235694 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.235698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.235834 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.235865 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.235918 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.235990 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"spinlock_wait_cycles":1280,"thread_start_us":105,"threads_started":1}
I20260923 14:45:33.236621 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.230359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.30.2.194:41029)
0923 14:45:33.230717 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.230732 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.231053 (+ 321us) server_negotiation.cc:407] Connection header received
0923 14:45:33.231295 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.231302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.231540 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.231701 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:33.232222 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.233407 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.234835 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.235595 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.235676 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.235867 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.236379 (+ 512us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.236483 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260923 14:45:33.236997 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.230848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.30.2.193:40767)
0923 14:45:33.231301 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.231308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.231331 (+ 23us) server_negotiation.cc:407] Connection header received
0923 14:45:33.231413 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.231420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.231660 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.232252 (+ 592us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:33.232389 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.233093 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.234733 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.235529 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.236542 (+ 1013us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.236730 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.236785 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.236875 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":87,"threads_started":1}
I20260923 14:45:33.237020 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.230187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41029 (local address 127.0.0.1:52356)
0923 14:45:33.230912 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.230967 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.231111 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.231918 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.231921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.231950 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:33.232093 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.232103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.233516 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.233519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.234714 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.234723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.235960 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.235963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.236047 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.236076 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.236835 (+ 759us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.236902 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":42,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260923 14:45:33.239872 429 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0ff39a85284bba9315c90304771aea (DEFAULT_TABLE table=test-table [id=66e1a2c4d1294b7d8471abd901b78156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:33.239874 32757 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0ff39a85284bba9315c90304771aea (DEFAULT_TABLE table=test-table [id=66e1a2c4d1294b7d8471abd901b78156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:33.241019 429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0ff39a85284bba9315c90304771aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:33.241017 32757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0ff39a85284bba9315c90304771aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:33.245966 32617 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0ff39a85284bba9315c90304771aea (DEFAULT_TABLE table=test-table [id=66e1a2c4d1294b7d8471abd901b78156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:33.247438 32617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0ff39a85284bba9315c90304771aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:33.250332 519 tablet_bootstrap.cc:492] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: Bootstrap starting.
I20260923 14:45:33.250778 518 tablet_bootstrap.cc:492] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: Bootstrap starting.
I20260923 14:45:33.253168 518 tablet_bootstrap.cc:654] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:33.253515 519 tablet_bootstrap.cc:654] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:33.254091 518 log.cc:824] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:33.254372 519 log.cc:824] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:33.255307 520 tablet_bootstrap.cc:492] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: Bootstrap starting.
I20260923 14:45:33.256455 519 tablet_bootstrap.cc:492] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: No bootstrap required, opened a new log
I20260923 14:45:33.256646 519 ts_tablet_manager.cc:1397] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:45:33.257244 518 tablet_bootstrap.cc:492] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: No bootstrap required, opened a new log
I20260923 14:45:33.257296 520 tablet_bootstrap.cc:654] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:33.257447 518 ts_tablet_manager.cc:1397] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 14:45:33.258086 520 log.cc:824] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:33.260020 520 tablet_bootstrap.cc:492] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: No bootstrap required, opened a new log
I20260923 14:45:33.260206 520 ts_tablet_manager.cc:1397] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:45:33.260567 519 raft_consensus.cc:348] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } }
I20260923 14:45:33.260641 518 raft_consensus.cc:348] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } }
I20260923 14:45:33.260833 519 raft_consensus.cc:374] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:33.260892 519 raft_consensus.cc:729] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be15d44699b45ba8e6dcf690a1280bf, State: Initialized, Role: FOLLOWER
I20260923 14:45:33.260900 518 raft_consensus.cc:374] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:33.260953 518 raft_consensus.cc:729] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dae3adade4a44a6b5efae6081091956, State: Initialized, Role: FOLLOWER
I20260923 14:45:33.261371 518 consensus_queue.cc:260] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } }
I20260923 14:45:33.261381 519 consensus_queue.cc:260] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } }
I20260923 14:45:33.262027 495 heartbeater.cc:503] Master 127.30.2.254:41293 was elected leader, sending a full tablet report...
I20260923 14:45:33.262344 518 ts_tablet_manager.cc:1428] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 14:45:33.262387 519 ts_tablet_manager.cc:1428] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 14:45:33.262692 355 heartbeater.cc:503] Master 127.30.2.254:41293 was elected leader, sending a full tablet report...
I20260923 14:45:33.265784 520 raft_consensus.cc:348] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } }
I20260923 14:45:33.266038 520 raft_consensus.cc:374] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:33.266101 520 raft_consensus.cc:729] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cef62d399340848985c5cc600c03f9, State: Initialized, Role: FOLLOWER
I20260923 14:45:33.266542 520 consensus_queue.cc:260] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } }
I20260923 14:45:33.267242 32683 heartbeater.cc:503] Master 127.30.2.254:41293 was elected leader, sending a full tablet report...
I20260923 14:45:33.267278 520 ts_tablet_manager.cc:1428] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260923 14:45:33.278508 525 raft_consensus.cc:482] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:33.278695 525 raft_consensus.cc:504] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } }
I20260923 14:45:33.279896 525 leader_election.cc:288] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dae3adade4a44a6b5efae6081091956 (127.30.2.195:33889), 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767)
I20260923 14:45:33.283917 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.280091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33889 (local address 127.30.2.194:58895)
0923 14:45:33.280216 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.280235 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.280333 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.280693 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.280697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.280724 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:33.280870 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.280877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.281709 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.281716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.282803 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.282814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.283419 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.283424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.283533 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.283561 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.283650 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.283727 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260923 14:45:33.283919 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.280243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:58895 (local address 127.30.2.195:33889)
0923 14:45:33.280374 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.280379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.280396 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:45:33.280469 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.280472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.280530 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.280606 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:33.280988 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.281611 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.282920 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.283304 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.283395 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.283434 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.283689 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.283772 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:45:33.284744 449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0ff39a85284bba9315c90304771aea" candidate_uuid: "6be15d44699b45ba8e6dcf690a1280bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dae3adade4a44a6b5efae6081091956" is_pre_election: true
I20260923 14:45:33.284863 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.280345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:45359 (local address 127.30.2.193:40767)
0923 14:45:33.280478 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.280482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.280777 (+ 295us) server_negotiation.cc:407] Connection header received
0923 14:45:33.280885 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.280887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.280942 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.281022 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:33.281776 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.282373 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.283691 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.284310 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.284426 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.284476 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.284678 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.284748 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:45:33.284878 527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.280173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:40767 (local address 127.30.2.194:45359)
0923 14:45:33.280677 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.280692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.280801 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.281043 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.281045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.281060 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:33.281630 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.281640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.282491 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.282494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.283562 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.283570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.284440 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.284444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.284552 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.284569 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.284660 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.284726 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":152,"threads_started":1}
I20260923 14:45:33.285089 449 raft_consensus.cc:2454] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6be15d44699b45ba8e6dcf690a1280bf in term 0.
I20260923 14:45:33.285502 32637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0ff39a85284bba9315c90304771aea" candidate_uuid: "6be15d44699b45ba8e6dcf690a1280bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cef62d399340848985c5cc600c03f9" is_pre_election: true
I20260923 14:45:33.285641 32710 leader_election.cc:302] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6be15d44699b45ba8e6dcf690a1280bf, 8dae3adade4a44a6b5efae6081091956; no voters:
I20260923 14:45:33.285802 32637 raft_consensus.cc:2454] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6be15d44699b45ba8e6dcf690a1280bf in term 0.
I20260923 14:45:33.285971 525 raft_consensus.cc:2781] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:33.286067 525 raft_consensus.cc:482] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:33.286127 525 raft_consensus.cc:3037] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:33.287160 525 raft_consensus.cc:504] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } }
I20260923 14:45:33.287523 525 leader_election.cc:288] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [CANDIDATE]: Term 1 election: Requested vote from peers 8dae3adade4a44a6b5efae6081091956 (127.30.2.195:33889), 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767)
I20260923 14:45:33.287875 32637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0ff39a85284bba9315c90304771aea" candidate_uuid: "6be15d44699b45ba8e6dcf690a1280bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cef62d399340848985c5cc600c03f9"
I20260923 14:45:33.287928 449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0ff39a85284bba9315c90304771aea" candidate_uuid: "6be15d44699b45ba8e6dcf690a1280bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dae3adade4a44a6b5efae6081091956"
I20260923 14:45:33.287999 32637 raft_consensus.cc:3037] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:33.288033 449 raft_consensus.cc:3037] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:33.289286 449 raft_consensus.cc:2454] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be15d44699b45ba8e6dcf690a1280bf in term 1.
I20260923 14:45:33.289317 32637 raft_consensus.cc:2454] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be15d44699b45ba8e6dcf690a1280bf in term 1.
I20260923 14:45:33.289573 32710 leader_election.cc:302] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6be15d44699b45ba8e6dcf690a1280bf, 8dae3adade4a44a6b5efae6081091956; no voters:
I20260923 14:45:33.289788 525 raft_consensus.cc:2781] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:33.289973 525 raft_consensus.cc:686] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 1 LEADER]: Becoming Leader. State: Replica: 6be15d44699b45ba8e6dcf690a1280bf, State: Running, Role: LEADER
I20260923 14:45:33.290324 525 consensus_queue.cc:237] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } }
I20260923 14:45:33.293431 32480 catalog_manager.cc:5495] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194). New cstate: current_term: 1 leader_uuid: "6be15d44699b45ba8e6dcf690a1280bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } health_report { overall_health: UNKNOWN } } }
Sep 23 14:45:33 dist-test-slave-wjf2 krb5kdc[32435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174732, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:33.342150 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.333722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41293 (local address 127.0.0.1:45766)
0923 14:45:33.334057 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.334070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.334179 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.334415 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.334418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.334821 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:33.334915 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.334921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.335658 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.335662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.336363 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.336372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.336787 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.336791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.336859 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.337424 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:33.337436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:33.338992 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:33.340565 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:33.340578 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:33.340584 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:33.340809 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:33.341068 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:33.341073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:33.341075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:33.341148 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:33.341518 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:33.341526 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:33.341631 (+ 105us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.341742 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.341912 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":138,"threads_started":1}
I20260923 14:45:33.342149 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.333771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.30.2.254:41293)
0923 14:45:33.333922 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.333928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.334170 (+ 242us) server_negotiation.cc:407] Connection header received
0923 14:45:33.334261 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.334263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.334304 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.334376 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:33.335017 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.335530 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.336465 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.336683 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.336759 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.339122 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:33.339125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:33.339127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:33.339146 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:33.340456 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:33.340904 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:33.340909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:33.340911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:33.340980 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:33.341243 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:33.341246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:33.341248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:33.341408 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:33.341517 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.341799 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.341968 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":53}
I20260923 14:45:33.346119 32480 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:45766:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 14:45:33.346927 32480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:33.351954 32617 tablet_service.cc:1467] Processing CreateTablet for tablet d5d12acfc9fe4b5ea13b0c0348e35cb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=039e578938434af3bb257c48fef00168]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:45:33.352172 32757 tablet_service.cc:1467] Processing CreateTablet for tablet d5d12acfc9fe4b5ea13b0c0348e35cb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=039e578938434af3bb257c48fef00168]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:45:33.352712 32617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d12acfc9fe4b5ea13b0c0348e35cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:33.352703 32757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d12acfc9fe4b5ea13b0c0348e35cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:33.352766 429 tablet_service.cc:1467] Processing CreateTablet for tablet d5d12acfc9fe4b5ea13b0c0348e35cb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=039e578938434af3bb257c48fef00168]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:45:33.353369 429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d12acfc9fe4b5ea13b0c0348e35cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:33.355675 520 tablet_bootstrap.cc:492] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9: Bootstrap starting.
I20260923 14:45:33.356175 518 tablet_bootstrap.cc:492] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: Bootstrap starting.
I20260923 14:45:33.356789 520 tablet_bootstrap.cc:654] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:33.356966 518 tablet_bootstrap.cc:654] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:33.357492 519 tablet_bootstrap.cc:492] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf: Bootstrap starting.
I20260923 14:45:33.358122 520 tablet_bootstrap.cc:492] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9: No bootstrap required, opened a new log
I20260923 14:45:33.358182 520 ts_tablet_manager.cc:1397] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 14:45:33.358429 519 tablet_bootstrap.cc:654] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:33.358533 520 raft_consensus.cc:348] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.358613 520 raft_consensus.cc:374] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:33.358633 520 raft_consensus.cc:729] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cef62d399340848985c5cc600c03f9, State: Initialized, Role: FOLLOWER
I20260923 14:45:33.358767 520 consensus_queue.cc:260] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.359026 520 ts_tablet_manager.cc:1428] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 14:45:33.359074 526 tablet_replica.cc:439] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } }
I20260923 14:45:33.359267 526 tablet_replica.cc:442] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:33.359712 519 tablet_bootstrap.cc:492] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf: No bootstrap required, opened a new log
I20260923 14:45:33.359841 519 ts_tablet_manager.cc:1397] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:33.359977 518 tablet_bootstrap.cc:492] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: No bootstrap required, opened a new log
I20260923 14:45:33.360064 518 ts_tablet_manager.cc:1397] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 14:45:33.360368 519 raft_consensus.cc:348] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.360502 519 raft_consensus.cc:374] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:33.360543 519 raft_consensus.cc:729] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be15d44699b45ba8e6dcf690a1280bf, State: Initialized, Role: FOLLOWER
I20260923 14:45:33.360502 518 raft_consensus.cc:348] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.360600 518 raft_consensus.cc:374] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:33.360634 518 raft_consensus.cc:729] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dae3adade4a44a6b5efae6081091956, State: Initialized, Role: FOLLOWER
I20260923 14:45:33.360670 519 consensus_queue.cc:260] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.360759 518 consensus_queue.cc:260] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.360940 519 ts_tablet_manager.cc:1428] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:33.361006 518 ts_tablet_manager.cc:1428] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:33.361182 525 tablet_replica.cc:439] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } }
I20260923 14:45:33.361217 524 tablet_replica.cc:439] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } }
I20260923 14:45:33.361328 525 tablet_replica.cc:442] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:33.361328 524 tablet_replica.cc:442] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:33.397526 524 raft_consensus.cc:482] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:33.397670 524 raft_consensus.cc:504] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.398802 524 leader_election.cc:288] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767), 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194:41029)
I20260923 14:45:33.402796 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.399126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:37855 (local address 127.30.2.193:40767)
0923 14:45:33.399277 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.399282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.399297 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:45:33.399351 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.399354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.399398 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.399458 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:33.399803 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.400633 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.401777 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.402182 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.402309 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.402363 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.402570 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.402665 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:45:33.402801 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.399001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:40767 (local address 127.30.2.195:37855)
0923 14:45:33.399133 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.399148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.399240 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.399502 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.399508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.399529 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:33.399677 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.399688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.400766 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.400770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.401655 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.401665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.402298 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.402301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.402414 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.402458 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.402536 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.402622 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":38,"spinlock_wait_cycles":1408}
I20260923 14:45:33.402837 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.399111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:45209 (local address 127.30.2.194:41029)
0923 14:45:33.399226 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.399230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.399560 (+ 330us) server_negotiation.cc:407] Connection header received
0923 14:45:33.399662 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.399664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.399709 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.399828 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:33.400053 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.400770 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.401785 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.402200 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.402328 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.402378 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.402637 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.402729 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:45:33.403168 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.399001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41029 (local address 127.30.2.195:45209)
0923 14:45:33.399455 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.399469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.399575 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.399839 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.399841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.399859 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:33.399953 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.399958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.400857 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.400859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.401666 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.401675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.402328 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.402332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.402448 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.402469 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.402601 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.402665 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":137,"spinlock_wait_cycles":3200,"thread_start_us":118,"threads_started":1}
I20260923 14:45:33.403337 32637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d12acfc9fe4b5ea13b0c0348e35cb8" candidate_uuid: "8dae3adade4a44a6b5efae6081091956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cef62d399340848985c5cc600c03f9" is_pre_election: true
I20260923 14:45:33.403489 32637 raft_consensus.cc:2454] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dae3adade4a44a6b5efae6081091956 in term 0.
I20260923 14:45:33.403692 309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d12acfc9fe4b5ea13b0c0348e35cb8" candidate_uuid: "8dae3adade4a44a6b5efae6081091956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6be15d44699b45ba8e6dcf690a1280bf" is_pre_election: true
I20260923 14:45:33.403934 309 raft_consensus.cc:2454] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dae3adade4a44a6b5efae6081091956 in term 0.
I20260923 14:45:33.403919 381 leader_election.cc:302] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cef62d399340848985c5cc600c03f9, 8dae3adade4a44a6b5efae6081091956; no voters:
I20260923 14:45:33.404176 524 raft_consensus.cc:2781] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:33.404248 524 raft_consensus.cc:482] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:33.404302 524 raft_consensus.cc:3037] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:33.405342 524 raft_consensus.cc:504] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.405742 524 leader_election.cc:288] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [CANDIDATE]: Term 1 election: Requested vote from peers 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767), 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194:41029)
I20260923 14:45:33.406095 32637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d12acfc9fe4b5ea13b0c0348e35cb8" candidate_uuid: "8dae3adade4a44a6b5efae6081091956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cef62d399340848985c5cc600c03f9"
I20260923 14:45:33.406204 32637 raft_consensus.cc:3037] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:33.406152 309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d12acfc9fe4b5ea13b0c0348e35cb8" candidate_uuid: "8dae3adade4a44a6b5efae6081091956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6be15d44699b45ba8e6dcf690a1280bf"
I20260923 14:45:33.406318 309 raft_consensus.cc:3037] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:33.407132 32637 raft_consensus.cc:2454] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dae3adade4a44a6b5efae6081091956 in term 1.
I20260923 14:45:33.407409 381 leader_election.cc:302] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cef62d399340848985c5cc600c03f9, 8dae3adade4a44a6b5efae6081091956; no voters:
I20260923 14:45:33.407444 309 raft_consensus.cc:2454] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dae3adade4a44a6b5efae6081091956 in term 1.
I20260923 14:45:33.407557 524 raft_consensus.cc:2781] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:33.407758 524 raft_consensus.cc:686] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [term 1 LEADER]: Becoming Leader. State: Replica: 8dae3adade4a44a6b5efae6081091956, State: Running, Role: LEADER
I20260923 14:45:33.408150 524 consensus_queue.cc:237] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.409826 524 tablet_replica.cc:439] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: TxnStatusTablet state changed. Reason: New leader 8dae3adade4a44a6b5efae6081091956. Latest consensus state: current_term: 1 leader_uuid: "8dae3adade4a44a6b5efae6081091956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } }
I20260923 14:45:33.409935 524 tablet_replica.cc:442] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:45:33.410208 542 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 14:45:33.410421 542 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 14:45:33.411242 32478 catalog_manager.cc:5495] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dae3adade4a44a6b5efae6081091956 (127.30.2.195). New cstate: current_term: 1 leader_uuid: "8dae3adade4a44a6b5efae6081091956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } health_report { overall_health: HEALTHY } } }
W20260923 14:45:33.425861 496 tablet.cc:2369] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:45:33.434278 356 tablet.cc:2369] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:45:33.438540 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.434542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42826 (local address 127.30.2.195:33889)
0923 14:45:33.434743 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.434748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.434763 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:45:33.434965 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.434968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.435021 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.435119 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:33.435371 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.436085 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.436991 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.437302 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.437407 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.437740 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.438143 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.438262 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.438361 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.438410 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260923 14:45:33.438540 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.434425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33889 (local address 127.0.0.1:42826)
0923 14:45:33.434678 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.434694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.434802 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.435132 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.435135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.435143 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:45:33.435262 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.435268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.436220 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.436227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.436871 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.436880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.437417 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.437421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.437532 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.437576 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.438237 (+ 661us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.438254 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.438345 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.438429 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260923 14:45:33.443997 32637 raft_consensus.cc:1261] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Refusing update from remote peer 8dae3adade4a44a6b5efae6081091956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:33.443992 309 raft_consensus.cc:1261] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Refusing update from remote peer 8dae3adade4a44a6b5efae6081091956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:33.444629 524 consensus_queue.cc:1035] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:33.444808 540 consensus_queue.cc:1035] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:33.447700 525 tablet_replica.cc:439] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf: TxnStatusTablet state changed. Reason: New leader 8dae3adade4a44a6b5efae6081091956. Latest consensus state: current_term: 1 leader_uuid: "8dae3adade4a44a6b5efae6081091956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } }
I20260923 14:45:33.447837 525 tablet_replica.cc:442] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:33.449829 526 tablet_replica.cc:439] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9: TxnStatusTablet state changed. Reason: New leader 8dae3adade4a44a6b5efae6081091956. Latest consensus state: current_term: 1 leader_uuid: "8dae3adade4a44a6b5efae6081091956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } }
I20260923 14:45:33.449954 526 tablet_replica.cc:442] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:33.452278 524 tablet_replica.cc:439] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8dae3adade4a44a6b5efae6081091956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } }
I20260923 14:45:33.452379 524 tablet_replica.cc:442] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:45:33.452508 524 tablet_replica.cc:439] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8dae3adade4a44a6b5efae6081091956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } }
I20260923 14:45:33.452580 524 tablet_replica.cc:442] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 8dae3adade4a44a6b5efae6081091956: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:45:33.452673 526 tablet_replica.cc:439] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8dae3adade4a44a6b5efae6081091956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } }
I20260923 14:45:33.452772 526 tablet_replica.cc:442] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 38cef62d399340848985c5cc600c03f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:33.453042 525 tablet_replica.cc:439] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8dae3adade4a44a6b5efae6081091956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } } }
I20260923 14:45:33.453158 525 tablet_replica.cc:442] T d5d12acfc9fe4b5ea13b0c0348e35cb8 P 6be15d44699b45ba8e6dcf690a1280bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:45:33.454927 545 mvcc.cc:205] Tried to move back new op lower bound from 7332555708179517440 to 7332555708042825728. Current Snapshot: MvccSnapshot[applied={T|T < 7332555708179517440}]
I20260923 14:45:33.455164 548 mvcc.cc:205] Tried to move back new op lower bound from 7332555708179517440 to 7332555708042825728. Current Snapshot: MvccSnapshot[applied={T|T < 7332555708179517440}]
I20260923 14:45:33.462762 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.459407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.30.2.194:41029)
0923 14:45:33.459542 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.459549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.459564 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:45:33.459649 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.459652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.459702 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.459817 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:33.460057 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.460563 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.461417 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.461658 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.461767 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.462027 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.462386 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.462503 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.462575 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.462624 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 14:45:33.463439 32684 tablet.cc:2369] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:45:33.472868 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.469762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:36553 (local address 127.30.2.195:33889)
0923 14:45:33.469870 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.469875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.470044 (+ 169us) server_negotiation.cc:407] Connection header received
0923 14:45:33.470199 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.470201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.470240 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.470314 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:33.470537 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.471002 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.471838 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.472034 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.472126 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.472428 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.472490 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.472564 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.472671 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.472739 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:45:33.472882 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.469672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33889 (local address 127.30.2.194:36553)
0923 14:45:33.469974 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.469984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.470085 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.470326 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.470329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.470337 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:45:33.470436 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.470448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.471107 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.471111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.471714 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.471721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.472141 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.472145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.472229 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.472271 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.472571 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.472580 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.472657 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.472747 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":92,"threads_started":1}
I20260923 14:45:33.479135 32637 raft_consensus.cc:1261] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Refusing update from remote peer 6be15d44699b45ba8e6dcf690a1280bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:33.479238 449 raft_consensus.cc:1261] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Refusing update from remote peer 6be15d44699b45ba8e6dcf690a1280bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:33.479583 528 consensus_queue.cc:1035] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:33.479725 525 consensus_queue.cc:1035] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:33.502449 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.498626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:42499 (local address 127.30.2.194:41029)
0923 14:45:33.498741 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.498745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.498778 (+ 33us) server_negotiation.cc:407] Connection header received
0923 14:45:33.498884 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.498886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.498924 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.499009 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:33.499400 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.500271 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.501447 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.501658 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.501737 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.502032 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.502094 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.502181 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.502299 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.502342 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:45:33.502497 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.498559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41029 (local address 127.30.2.195:42499)
0923 14:45:33.498713 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.498728 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.498820 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.499022 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.499025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.499033 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:45:33.499156 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.499164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.500407 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.500411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.501340 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.501351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.501765 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.501768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.501861 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.501915 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.502210 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.502216 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.502283 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.502364 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 14:45:33.529940 32479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:33.530148 32479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:33.532997 32479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150: Sending DeleteTablet for 3 replicas of tablet 6f0ff39a85284bba9315c90304771aea
I20260923 14:45:33.533811 32757 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f0ff39a85284bba9315c90304771aea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:33 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:52356
I20260923 14:45:33.533898 32617 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f0ff39a85284bba9315c90304771aea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:33 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:45366
I20260923 14:45:33.534539 429 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f0ff39a85284bba9315c90304771aea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:33 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42810
I20260923 14:45:33.535187 564 tablet_replica.cc:330] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: stopping tablet replica
I20260923 14:45:33.535178 32479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 14:45:33.535493 564 raft_consensus.cc:2229] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 1 LEADER]: Raft consensus shutting down.
W20260923 14:45:33.535578 32479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:33.535868 564 raft_consensus.cc:2258] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:33.536247 565 tablet_replica.cc:330] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: stopping tablet replica
I20260923 14:45:33.536532 565 raft_consensus.cc:2229] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:33.536743 565 raft_consensus.cc:2258] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:33.538615 564 ts_tablet_manager.cc:1905] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:33.540572 32617 tablet_service.cc:1467] Processing CreateTablet for tablet d68d878d0d01483cbf68986fd71e4aed (DEFAULT_TABLE table=test-table [id=b1458d06891c4ce8b1d8757f23ba4172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:33.540771 32757 tablet_service.cc:1467] Processing CreateTablet for tablet d68d878d0d01483cbf68986fd71e4aed (DEFAULT_TABLE table=test-table [id=b1458d06891c4ce8b1d8757f23ba4172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:33.540855 32617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68d878d0d01483cbf68986fd71e4aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:33.540999 32757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68d878d0d01483cbf68986fd71e4aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:33.542415 566 tablet_replica.cc:330] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: stopping tablet replica
I20260923 14:45:33.542685 566 raft_consensus.cc:2229] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:33.542831 566 raft_consensus.cc:2258] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:33.543414 520 tablet_bootstrap.cc:492] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9: Bootstrap starting.
I20260923 14:45:33.543761 519 tablet_bootstrap.cc:492] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf: Bootstrap starting.
I20260923 14:45:33.544409 520 tablet_bootstrap.cc:654] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:33.544595 566 ts_tablet_manager.cc:1905] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:33.544690 519 tablet_bootstrap.cc:654] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:33.545825 565 ts_tablet_manager.cc:1905] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:33.547052 429 tablet_service.cc:1467] Processing CreateTablet for tablet d68d878d0d01483cbf68986fd71e4aed (DEFAULT_TABLE table=test-table [id=b1458d06891c4ce8b1d8757f23ba4172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:33.547322 429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68d878d0d01483cbf68986fd71e4aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:33.547624 566 ts_tablet_manager.cc:1918] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:45:33.547714 566 log.cc:1196] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/6f0ff39a85284bba9315c90304771aea
I20260923 14:45:33.548547 565 ts_tablet_manager.cc:1918] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:45:33.548636 565 log.cc:1196] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/6f0ff39a85284bba9315c90304771aea
I20260923 14:45:33.548934 565 ts_tablet_manager.cc:1939] T 6f0ff39a85284bba9315c90304771aea P 38cef62d399340848985c5cc600c03f9: Deleting consensus metadata
I20260923 14:45:33.550141 32465 catalog_manager.cc:4841] TS 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767): tablet 6f0ff39a85284bba9315c90304771aea (table test-table [id=66e1a2c4d1294b7d8471abd901b78156]) successfully deleted
I20260923 14:45:33.550374 518 tablet_bootstrap.cc:492] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956: Bootstrap starting.
I20260923 14:45:33.551283 518 tablet_bootstrap.cc:654] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:33.552254 519 tablet_bootstrap.cc:492] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf: No bootstrap required, opened a new log
I20260923 14:45:33.552346 519 ts_tablet_manager.cc:1397] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260923 14:45:33.552806 519 raft_consensus.cc:348] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.552932 519 raft_consensus.cc:374] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:33.552974 519 raft_consensus.cc:729] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be15d44699b45ba8e6dcf690a1280bf, State: Initialized, Role: FOLLOWER
I20260923 14:45:33.553118 519 consensus_queue.cc:260] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.553393 519 ts_tablet_manager.cc:1428] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 14:45:33.554234 520 tablet_bootstrap.cc:492] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9: No bootstrap required, opened a new log
I20260923 14:45:33.554332 520 ts_tablet_manager.cc:1397] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260923 14:45:33.554773 520 raft_consensus.cc:348] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.554890 520 raft_consensus.cc:374] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:33.554937 520 raft_consensus.cc:729] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cef62d399340848985c5cc600c03f9, State: Initialized, Role: FOLLOWER
I20260923 14:45:33.555053 520 consensus_queue.cc:260] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.555338 520 ts_tablet_manager.cc:1428] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:33.556094 566 ts_tablet_manager.cc:1939] T 6f0ff39a85284bba9315c90304771aea P 8dae3adade4a44a6b5efae6081091956: Deleting consensus metadata
I20260923 14:45:33.557237 32466 catalog_manager.cc:4841] TS 8dae3adade4a44a6b5efae6081091956 (127.30.2.195:33889): tablet 6f0ff39a85284bba9315c90304771aea (table test-table [id=66e1a2c4d1294b7d8471abd901b78156]) successfully deleted
I20260923 14:45:33.557648 564 ts_tablet_manager.cc:1918] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:45:33.557754 564 log.cc:1196] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/6f0ff39a85284bba9315c90304771aea
I20260923 14:45:33.558082 564 ts_tablet_manager.cc:1939] T 6f0ff39a85284bba9315c90304771aea P 6be15d44699b45ba8e6dcf690a1280bf: Deleting consensus metadata
I20260923 14:45:33.558130 518 tablet_bootstrap.cc:492] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956: No bootstrap required, opened a new log
I20260923 14:45:33.558208 518 ts_tablet_manager.cc:1397] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260923 14:45:33.558686 518 raft_consensus.cc:348] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.558796 518 raft_consensus.cc:374] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:33.558835 518 raft_consensus.cc:729] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dae3adade4a44a6b5efae6081091956, State: Initialized, Role: FOLLOWER
I20260923 14:45:33.558970 518 consensus_queue.cc:260] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.559167 32466 catalog_manager.cc:4841] TS 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194:41029): tablet 6f0ff39a85284bba9315c90304771aea (table test-table [id=66e1a2c4d1294b7d8471abd901b78156]) successfully deleted
I20260923 14:45:33.559248 518 ts_tablet_manager.cc:1428] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:33.752314 526 raft_consensus.cc:482] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:33.752487 526 raft_consensus.cc:504] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.753667 526 leader_election.cc:288] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194:41029), 8dae3adade4a44a6b5efae6081091956 (127.30.2.195:33889)
I20260923 14:45:33.757931 571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.754231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33889 (local address 127.30.2.193:33909)
0923 14:45:33.754521 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.754536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.754633 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.754879 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.754881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.754914 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:33.755010 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.755014 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.755886 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.755892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.756783 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.756793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.757495 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.757501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.757608 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.757625 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.757683 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.757759 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":41,"spinlock_wait_cycles":768,"thread_start_us":119,"threads_started":1}
I20260923 14:45:33.757941 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.754389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:33909 (local address 127.30.2.195:33889)
0923 14:45:33.754530 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.754535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.754609 (+ 74us) server_negotiation.cc:407] Connection header received
0923 14:45:33.754714 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.754717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.754772 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.754855 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:33.755146 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.755707 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.756938 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.757346 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.757494 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.757548 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.757727 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.757801 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:45:33.757966 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.753863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41029 (local address 127.30.2.193:51463)
0923 14:45:33.754161 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.754195 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.754340 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.754841 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.754847 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.754866 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:33.754998 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.755004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.755756 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.755761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.756772 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.756782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.757439 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.757447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.757549 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.757653 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.757747 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.757815 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":76,"spinlock_wait_cycles":15744,"thread_start_us":85,"threads_started":1}
I20260923 14:45:33.757977 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.754018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:51463 (local address 127.30.2.194:41029)
0923 14:45:33.754185 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.754191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.754283 (+ 92us) server_negotiation.cc:407] Connection header received
0923 14:45:33.754525 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.754530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.754606 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.754666 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:33.755127 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.755636 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.756895 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.757310 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.757400 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.757480 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.757787 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.757863 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:45:33.758450 449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68d878d0d01483cbf68986fd71e4aed" candidate_uuid: "38cef62d399340848985c5cc600c03f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dae3adade4a44a6b5efae6081091956" is_pre_election: true
I20260923 14:45:33.758625 449 raft_consensus.cc:2454] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38cef62d399340848985c5cc600c03f9 in term 0.
I20260923 14:45:33.758750 309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68d878d0d01483cbf68986fd71e4aed" candidate_uuid: "38cef62d399340848985c5cc600c03f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6be15d44699b45ba8e6dcf690a1280bf" is_pre_election: true
I20260923 14:45:33.758898 309 raft_consensus.cc:2454] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38cef62d399340848985c5cc600c03f9 in term 0.
I20260923 14:45:33.759545 32570 leader_election.cc:302] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cef62d399340848985c5cc600c03f9, 8dae3adade4a44a6b5efae6081091956; no voters:
I20260923 14:45:33.759821 526 raft_consensus.cc:2781] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:33.759881 526 raft_consensus.cc:482] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:33.759910 526 raft_consensus.cc:3037] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:33.760841 526 raft_consensus.cc:504] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.761116 526 leader_election.cc:288] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [CANDIDATE]: Term 1 election: Requested vote from peers 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194:41029), 8dae3adade4a44a6b5efae6081091956 (127.30.2.195:33889)
I20260923 14:45:33.761555 449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68d878d0d01483cbf68986fd71e4aed" candidate_uuid: "38cef62d399340848985c5cc600c03f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dae3adade4a44a6b5efae6081091956"
I20260923 14:45:33.761554 309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68d878d0d01483cbf68986fd71e4aed" candidate_uuid: "38cef62d399340848985c5cc600c03f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6be15d44699b45ba8e6dcf690a1280bf"
I20260923 14:45:33.761685 449 raft_consensus.cc:3037] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:33.761685 309 raft_consensus.cc:3037] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:33.762902 449 raft_consensus.cc:2454] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38cef62d399340848985c5cc600c03f9 in term 1.
I20260923 14:45:33.762902 309 raft_consensus.cc:2454] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38cef62d399340848985c5cc600c03f9 in term 1.
I20260923 14:45:33.763195 32570 leader_election.cc:302] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cef62d399340848985c5cc600c03f9, 8dae3adade4a44a6b5efae6081091956; no voters:
I20260923 14:45:33.763391 526 raft_consensus.cc:2781] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:33.763624 526 raft_consensus.cc:686] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 1 LEADER]: Becoming Leader. State: Replica: 38cef62d399340848985c5cc600c03f9, State: Running, Role: LEADER
I20260923 14:45:33.763934 526 consensus_queue.cc:237] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:33.766444 32479 catalog_manager.cc:5495] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38cef62d399340848985c5cc600c03f9 (127.30.2.193). New cstate: current_term: 1 leader_uuid: "38cef62d399340848985c5cc600c03f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20260923 14:45:33.796564 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.792972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45374 (local address 127.30.2.193:40767)
0923 14:45:33.793118 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.793123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.793137 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:45:33.793318 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.793320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.793377 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.793472 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:33.793702 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.794251 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.795036 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.795279 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.795364 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.795577 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.796156 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.796271 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.796332 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.796383 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 14:45:33.807206 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.803953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:49025 (local address 127.30.2.195:33889)
0923 14:45:33.804081 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.804115 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.804240 (+ 125us) server_negotiation.cc:407] Connection header received
0923 14:45:33.804340 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.804343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.804386 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.804454 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:33.804704 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.805221 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.806175 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.806388 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.806467 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.806758 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.806833 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.806885 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.807050 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.807099 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:45:33.807246 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.803832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33889 (local address 127.30.2.193:49025)
0923 14:45:33.804153 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.804167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.804261 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.804473 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.804476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.804482 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:45:33.804584 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.804588 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.805334 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.805336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.806054 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.806064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.806507 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.806511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.806593 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.806641 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.806948 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.806956 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.807032 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.807109 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260923 14:45:33.813858 309 raft_consensus.cc:1261] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Refusing update from remote peer 38cef62d399340848985c5cc600c03f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:33.813930 449 raft_consensus.cc:1261] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Refusing update from remote peer 38cef62d399340848985c5cc600c03f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:33.814392 572 consensus_queue.cc:1035] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:33.814611 526 consensus_queue.cc:1035] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:33.833523 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.829802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:40767 (local address 127.30.2.195:43733)
0923 14:45:33.830016 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:33.830035 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:33.830126 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:33.830379 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:33.830382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:33.830390 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:45:33.830497 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.830504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.831366 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.831369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.832039 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:33.832049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.832733 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.832736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:33.832845 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.832881 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.833224 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.833233 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 14:45:33.833308 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 14:45:33.833405 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260923 14:45:33.833555 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:33.829865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:43733 (local address 127.30.2.193:40767)
0923 14:45:33.829989 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:33.829993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:33.830087 (+ 94us) server_negotiation.cc:407] Connection header received
0923 14:45:33.830237 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:33.830240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:33.830290 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:33.830366 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:33.830608 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.831254 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.832205 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:33.832564 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:33.832742 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:33.833008 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:33.833110 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:33.833194 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:33.833330 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 14:45:33.833406 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 14:45:33.834286 548 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 14:45:35.402626 32480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:35.402783 32480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:35.405540 32480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150: Sending DeleteTablet for 3 replicas of tablet d68d878d0d01483cbf68986fd71e4aed
I20260923 14:45:35.406180 32617 tablet_service.cc:1514] Processing DeleteTablet for tablet d68d878d0d01483cbf68986fd71e4aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:35 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:45366
I20260923 14:45:35.406334 32757 tablet_service.cc:1514] Processing DeleteTablet for tablet d68d878d0d01483cbf68986fd71e4aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:35 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:52356
I20260923 14:45:35.406347 429 tablet_service.cc:1514] Processing DeleteTablet for tablet d68d878d0d01483cbf68986fd71e4aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:35 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42810
I20260923 14:45:35.406569 605 tablet_replica.cc:330] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9: stopping tablet replica
I20260923 14:45:35.406719 605 raft_consensus.cc:2229] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:45:35.406858 606 tablet_replica.cc:330] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956: stopping tablet replica
I20260923 14:45:35.406983 606 raft_consensus.cc:2229] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:35.407096 606 raft_consensus.cc:2258] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:35.407099 605 raft_consensus.cc:2258] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:35.407375 607 tablet_replica.cc:330] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf: stopping tablet replica
I20260923 14:45:35.407507 607 raft_consensus.cc:2229] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:35.407634 607 raft_consensus.cc:2258] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:35.408248 605 ts_tablet_manager.cc:1905] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:35.408248 606 ts_tablet_manager.cc:1905] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:35.411145 606 ts_tablet_manager.cc:1918] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 14:45:35.411242 606 log.cc:1196] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/d68d878d0d01483cbf68986fd71e4aed
I20260923 14:45:35.411397 605 ts_tablet_manager.cc:1918] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 14:45:35.411527 605 log.cc:1196] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/d68d878d0d01483cbf68986fd71e4aed
I20260923 14:45:35.411556 606 ts_tablet_manager.cc:1939] T d68d878d0d01483cbf68986fd71e4aed P 8dae3adade4a44a6b5efae6081091956: Deleting consensus metadata
I20260923 14:45:35.411877 605 ts_tablet_manager.cc:1939] T d68d878d0d01483cbf68986fd71e4aed P 38cef62d399340848985c5cc600c03f9: Deleting consensus metadata
I20260923 14:45:35.411978 607 ts_tablet_manager.cc:1905] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:35.412361 32466 catalog_manager.cc:4841] TS 8dae3adade4a44a6b5efae6081091956 (127.30.2.195:33889): tablet d68d878d0d01483cbf68986fd71e4aed (table test-table [id=b1458d06891c4ce8b1d8757f23ba4172]) successfully deleted
I20260923 14:45:35.412597 32465 catalog_manager.cc:4841] TS 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767): tablet d68d878d0d01483cbf68986fd71e4aed (table test-table [id=b1458d06891c4ce8b1d8757f23ba4172]) successfully deleted
I20260923 14:45:35.415185 607 ts_tablet_manager.cc:1918] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 14:45:35.415269 607 log.cc:1196] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/d68d878d0d01483cbf68986fd71e4aed
I20260923 14:45:35.415540 607 ts_tablet_manager.cc:1939] T d68d878d0d01483cbf68986fd71e4aed P 6be15d44699b45ba8e6dcf690a1280bf: Deleting consensus metadata
I20260923 14:45:35.416239 32466 catalog_manager.cc:4841] TS 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194:41029): tablet d68d878d0d01483cbf68986fd71e4aed (table test-table [id=b1458d06891c4ce8b1d8757f23ba4172]) successfully deleted
Sep 23 14:45:35 dist-test-slave-wjf2 krb5kdc[32435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 14:45:35 dist-test-slave-wjf2 krb5kdc[32435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:35.445055 615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:35.429295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.30.2.254:41293)
0923 14:45:35.429646 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:35.429653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:35.429667 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:45:35.429813 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:35.429816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:35.429862 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:35.429946 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:35.430698 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:35.431248 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:35.432475 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:35.432901 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:35.433027 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:35.441348 (+ 8321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:35.441351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:35.441353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:35.441382 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:35.443035 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:35.443594 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:35.443597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:35.443598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:35.443679 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:35.444015 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:35.444018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:35.444020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:35.444321 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:35.444471 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:35.444751 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 14:45:35.444881 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":128,"threads_started":1}
I20260923 14:45:35.448060 32480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:45:35.448635 32480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:35.453929 32757 tablet_service.cc:1467] Processing CreateTablet for tablet 354702136e454a079a8901f3c75ed5db (DEFAULT_TABLE table=test-table [id=32a0e62dac014a29ad715cd571f987c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:35.454013 429 tablet_service.cc:1467] Processing CreateTablet for tablet 354702136e454a079a8901f3c75ed5db (DEFAULT_TABLE table=test-table [id=32a0e62dac014a29ad715cd571f987c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:35.454022 32617 tablet_service.cc:1467] Processing CreateTablet for tablet 354702136e454a079a8901f3c75ed5db (DEFAULT_TABLE table=test-table [id=32a0e62dac014a29ad715cd571f987c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:35.454293 32757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354702136e454a079a8901f3c75ed5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:35.454344 429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354702136e454a079a8901f3c75ed5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:35.454401 32617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354702136e454a079a8901f3c75ed5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:35.456707 616 tablet_bootstrap.cc:492] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9: Bootstrap starting.
I20260923 14:45:35.457324 618 tablet_bootstrap.cc:492] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956: Bootstrap starting.
I20260923 14:45:35.457438 617 tablet_bootstrap.cc:492] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf: Bootstrap starting.
I20260923 14:45:35.457954 616 tablet_bootstrap.cc:654] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:35.458352 617 tablet_bootstrap.cc:654] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:35.458389 618 tablet_bootstrap.cc:654] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:35.459312 616 tablet_bootstrap.cc:492] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9: No bootstrap required, opened a new log
I20260923 14:45:35.459455 616 ts_tablet_manager.cc:1397] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 14:45:35.459596 617 tablet_bootstrap.cc:492] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf: No bootstrap required, opened a new log
I20260923 14:45:35.459686 618 tablet_bootstrap.cc:492] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956: No bootstrap required, opened a new log
I20260923 14:45:35.459714 617 ts_tablet_manager.cc:1397] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:45:35.459774 618 ts_tablet_manager.cc:1397] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 14:45:35.460006 616 raft_consensus.cc:348] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.460144 616 raft_consensus.cc:374] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:35.460189 616 raft_consensus.cc:729] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cef62d399340848985c5cc600c03f9, State: Initialized, Role: FOLLOWER
I20260923 14:45:35.460173 617 raft_consensus.cc:348] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.460235 618 raft_consensus.cc:348] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.460309 617 raft_consensus.cc:374] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:35.460345 618 raft_consensus.cc:374] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:35.460348 617 raft_consensus.cc:729] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be15d44699b45ba8e6dcf690a1280bf, State: Initialized, Role: FOLLOWER
I20260923 14:45:35.460391 618 raft_consensus.cc:729] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dae3adade4a44a6b5efae6081091956, State: Initialized, Role: FOLLOWER
I20260923 14:45:35.460372 616 consensus_queue.cc:260] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.460494 617 consensus_queue.cc:260] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.460520 618 consensus_queue.cc:260] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.460695 616 ts_tablet_manager.cc:1428] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:35.460834 618 ts_tablet_manager.cc:1428] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:35.460894 617 ts_tablet_manager.cc:1428] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 14:45:35.514899 581 raft_consensus.cc:482] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:35.515066 581 raft_consensus.cc:504] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.515806 581 leader_election.cc:288] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767), 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194:41029)
I20260923 14:45:35.516351 32637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354702136e454a079a8901f3c75ed5db" candidate_uuid: "8dae3adade4a44a6b5efae6081091956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cef62d399340848985c5cc600c03f9" is_pre_election: true
I20260923 14:45:35.516472 309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354702136e454a079a8901f3c75ed5db" candidate_uuid: "8dae3adade4a44a6b5efae6081091956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6be15d44699b45ba8e6dcf690a1280bf" is_pre_election: true
I20260923 14:45:35.516602 32637 raft_consensus.cc:2454] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dae3adade4a44a6b5efae6081091956 in term 0.
I20260923 14:45:35.516645 309 raft_consensus.cc:2454] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dae3adade4a44a6b5efae6081091956 in term 0.
I20260923 14:45:35.517062 382 leader_election.cc:302] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38cef62d399340848985c5cc600c03f9, 6be15d44699b45ba8e6dcf690a1280bf, 8dae3adade4a44a6b5efae6081091956; no voters:
I20260923 14:45:35.517249 581 raft_consensus.cc:2781] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:35.517310 581 raft_consensus.cc:482] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:35.517359 581 raft_consensus.cc:3037] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:35.518354 581 raft_consensus.cc:504] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.518716 581 leader_election.cc:288] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [CANDIDATE]: Term 1 election: Requested vote from peers 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767), 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194:41029)
I20260923 14:45:35.519058 32637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354702136e454a079a8901f3c75ed5db" candidate_uuid: "8dae3adade4a44a6b5efae6081091956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cef62d399340848985c5cc600c03f9"
I20260923 14:45:35.519212 32637 raft_consensus.cc:3037] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:35.519199 309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354702136e454a079a8901f3c75ed5db" candidate_uuid: "8dae3adade4a44a6b5efae6081091956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6be15d44699b45ba8e6dcf690a1280bf"
I20260923 14:45:35.519326 309 raft_consensus.cc:3037] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:35.520416 309 raft_consensus.cc:2454] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dae3adade4a44a6b5efae6081091956 in term 1.
I20260923 14:45:35.520416 32637 raft_consensus.cc:2454] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dae3adade4a44a6b5efae6081091956 in term 1.
I20260923 14:45:35.520725 382 leader_election.cc:302] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6be15d44699b45ba8e6dcf690a1280bf, 8dae3adade4a44a6b5efae6081091956; no voters:
I20260923 14:45:35.520948 581 raft_consensus.cc:2781] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:35.521099 581 raft_consensus.cc:686] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 1 LEADER]: Becoming Leader. State: Replica: 8dae3adade4a44a6b5efae6081091956, State: Running, Role: LEADER
I20260923 14:45:35.521301 581 consensus_queue.cc:237] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.523728 32479 catalog_manager.cc:5495] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dae3adade4a44a6b5efae6081091956 (127.30.2.195). New cstate: current_term: 1 leader_uuid: "8dae3adade4a44a6b5efae6081091956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } health_report { overall_health: HEALTHY } } }
I20260923 14:45:35.550827 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:35.547496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.30.2.195:33889)
0923 14:45:35.547827 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:35.547835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:35.547845 (+ 10us) server_negotiation.cc:407] Connection header received
0923 14:45:35.547881 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:35.547883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:35.547924 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:35.548017 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:35.548342 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:35.548864 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:35.549743 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:35.549965 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:35.550078 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:35.550363 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:35.550434 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:35.550515 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:35.550610 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 14:45:35.550662 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":118,"threads_started":1}
I20260923 14:45:35.561067 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:35.557992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:46265 (local address 127.30.2.195:33889)
0923 14:45:35.558133 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:35.558137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:35.558151 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:45:35.558270 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:35.558272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:35.558315 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:35.558382 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:35.558638 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:35.559206 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:35.560130 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:35.560355 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:35.560438 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:35.560682 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:35.560749 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:35.560809 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:35.560892 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 14:45:35.560933 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:45:35.561081 628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:35.557730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33889 (local address 127.30.2.195:46265)
0923 14:45:35.558033 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:35.558057 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:35.558169 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:35.558414 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:35.558416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:35.558423 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:45:35.558527 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:35.558533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:35.559325 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:35.559328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:35.559982 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:35.559991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:35.560461 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:35.560464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:35.560532 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:35.560557 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:45:35.560830 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:45:35.560835 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:45:35.560884 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 14:45:35.560953 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260923 14:45:35.566691 309 raft_consensus.cc:1261] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Refusing update from remote peer 8dae3adade4a44a6b5efae6081091956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:35.566712 32637 raft_consensus.cc:1261] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Refusing update from remote peer 8dae3adade4a44a6b5efae6081091956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:35.567167 581 consensus_queue.cc:1035] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:35.567315 600 consensus_queue.cc:1035] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:35.601389 32477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:35.601585 32477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:35.603899 32477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150: Sending DeleteTablet for 3 replicas of tablet 354702136e454a079a8901f3c75ed5db
I20260923 14:45:35.604471 32617 tablet_service.cc:1514] Processing DeleteTablet for tablet 354702136e454a079a8901f3c75ed5db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:35 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:45366
I20260923 14:45:35.604673 605 tablet_replica.cc:330] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9: stopping tablet replica
I20260923 14:45:35.604712 32757 tablet_service.cc:1514] Processing DeleteTablet for tablet 354702136e454a079a8901f3c75ed5db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:35 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:52356
I20260923 14:45:35.604794 605 raft_consensus.cc:2229] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:35.604768 428 tablet_service.cc:1514] Processing DeleteTablet for tablet 354702136e454a079a8901f3c75ed5db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:35 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42810
I20260923 14:45:35.604905 605 raft_consensus.cc:2258] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:35.604960 606 tablet_replica.cc:330] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956: stopping tablet replica
I20260923 14:45:35.605064 607 tablet_replica.cc:330] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf: stopping tablet replica
I20260923 14:45:35.605086 606 raft_consensus.cc:2229] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:45:35.605191 607 raft_consensus.cc:2229] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:35.605309 607 raft_consensus.cc:2258] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:35.605412 606 raft_consensus.cc:2258] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:35.605780 32477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:45:35.606216 32477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:35.606261 606 ts_tablet_manager.cc:1905] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:35.607259 605 ts_tablet_manager.cc:1905] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:35.609089 606 ts_tablet_manager.cc:1918] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:45:35.609169 606 log.cc:1196] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/354702136e454a079a8901f3c75ed5db
I20260923 14:45:35.609455 606 ts_tablet_manager.cc:1939] T 354702136e454a079a8901f3c75ed5db P 8dae3adade4a44a6b5efae6081091956: Deleting consensus metadata
I20260923 14:45:35.609616 605 ts_tablet_manager.cc:1918] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:45:35.609699 605 log.cc:1196] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/354702136e454a079a8901f3c75ed5db
I20260923 14:45:35.609923 605 ts_tablet_manager.cc:1939] T 354702136e454a079a8901f3c75ed5db P 38cef62d399340848985c5cc600c03f9: Deleting consensus metadata
I20260923 14:45:35.610106 32466 catalog_manager.cc:4841] TS 8dae3adade4a44a6b5efae6081091956 (127.30.2.195:33889): tablet 354702136e454a079a8901f3c75ed5db (table test-table [id=32a0e62dac014a29ad715cd571f987c0]) successfully deleted
I20260923 14:45:35.610564 32465 catalog_manager.cc:4841] TS 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767): tablet 354702136e454a079a8901f3c75ed5db (table test-table [id=32a0e62dac014a29ad715cd571f987c0]) successfully deleted
I20260923 14:45:35.610867 607 ts_tablet_manager.cc:1905] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:35.613531 607 ts_tablet_manager.cc:1918] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:45:35.613627 607 log.cc:1196] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/354702136e454a079a8901f3c75ed5db
I20260923 14:45:35.613860 607 ts_tablet_manager.cc:1939] T 354702136e454a079a8901f3c75ed5db P 6be15d44699b45ba8e6dcf690a1280bf: Deleting consensus metadata
I20260923 14:45:35.614578 32466 catalog_manager.cc:4841] TS 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194:41029): tablet 354702136e454a079a8901f3c75ed5db (table test-table [id=32a0e62dac014a29ad715cd571f987c0]) successfully deleted
I20260923 14:45:35.617432 32617 tablet_service.cc:1467] Processing CreateTablet for tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0 (DEFAULT_TABLE table=test-table [id=ce1069abc1144e31b47ab2d7946a16de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:35.617481 32757 tablet_service.cc:1467] Processing CreateTablet for tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0 (DEFAULT_TABLE table=test-table [id=ce1069abc1144e31b47ab2d7946a16de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:35.617714 32617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:35.617743 32757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:35.619629 617 tablet_bootstrap.cc:492] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf: Bootstrap starting.
I20260923 14:45:35.619987 616 tablet_bootstrap.cc:492] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9: Bootstrap starting.
I20260923 14:45:35.620278 428 tablet_service.cc:1467] Processing CreateTablet for tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0 (DEFAULT_TABLE table=test-table [id=ce1069abc1144e31b47ab2d7946a16de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:35.620502 617 tablet_bootstrap.cc:654] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:35.620548 428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:35.620780 616 tablet_bootstrap.cc:654] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:35.622027 616 tablet_bootstrap.cc:492] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9: No bootstrap required, opened a new log
I20260923 14:45:35.622112 616 ts_tablet_manager.cc:1397] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:45:35.622488 616 raft_consensus.cc:348] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.622598 616 raft_consensus.cc:374] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:35.622639 616 raft_consensus.cc:729] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cef62d399340848985c5cc600c03f9, State: Initialized, Role: FOLLOWER
I20260923 14:45:35.622762 616 consensus_queue.cc:260] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.622915 618 tablet_bootstrap.cc:492] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956: Bootstrap starting.
I20260923 14:45:35.623019 616 ts_tablet_manager.cc:1428] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:35.623651 618 tablet_bootstrap.cc:654] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:35.624871 618 tablet_bootstrap.cc:492] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956: No bootstrap required, opened a new log
I20260923 14:45:35.624965 618 ts_tablet_manager.cc:1397] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:45:35.625304 618 raft_consensus.cc:348] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.625413 618 raft_consensus.cc:374] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:35.625453 618 raft_consensus.cc:729] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dae3adade4a44a6b5efae6081091956, State: Initialized, Role: FOLLOWER
I20260923 14:45:35.625586 618 consensus_queue.cc:260] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.625887 618 ts_tablet_manager.cc:1428] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 14:45:35.626843 617 tablet_bootstrap.cc:492] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf: No bootstrap required, opened a new log
I20260923 14:45:35.626940 617 ts_tablet_manager.cc:1397] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260923 14:45:35.627372 617 raft_consensus.cc:348] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.627486 617 raft_consensus.cc:374] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:35.627517 617 raft_consensus.cc:729] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be15d44699b45ba8e6dcf690a1280bf, State: Initialized, Role: FOLLOWER
I20260923 14:45:35.627650 617 consensus_queue.cc:260] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.627928 617 ts_tablet_manager.cc:1428] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:35.742324 622 raft_consensus.cc:482] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:35.742499 622 raft_consensus.cc:504] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.742965 622 leader_election.cc:288] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767), 8dae3adade4a44a6b5efae6081091956 (127.30.2.195:33889)
I20260923 14:45:35.743520 32637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb49f0eb7b0a48ec8892d6f81ae5e0d0" candidate_uuid: "6be15d44699b45ba8e6dcf690a1280bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cef62d399340848985c5cc600c03f9" is_pre_election: true
I20260923 14:45:35.743518 449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb49f0eb7b0a48ec8892d6f81ae5e0d0" candidate_uuid: "6be15d44699b45ba8e6dcf690a1280bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dae3adade4a44a6b5efae6081091956" is_pre_election: true
I20260923 14:45:35.743680 32637 raft_consensus.cc:2454] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6be15d44699b45ba8e6dcf690a1280bf in term 0.
I20260923 14:45:35.743698 449 raft_consensus.cc:2454] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6be15d44699b45ba8e6dcf690a1280bf in term 0.
I20260923 14:45:35.744050 32709 leader_election.cc:302] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cef62d399340848985c5cc600c03f9, 6be15d44699b45ba8e6dcf690a1280bf; no voters:
I20260923 14:45:35.744247 622 raft_consensus.cc:2781] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:35.744310 622 raft_consensus.cc:482] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:35.744362 622 raft_consensus.cc:3037] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:35.745445 622 raft_consensus.cc:504] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.745841 622 leader_election.cc:288] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [CANDIDATE]: Term 1 election: Requested vote from peers 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767), 8dae3adade4a44a6b5efae6081091956 (127.30.2.195:33889)
I20260923 14:45:35.746263 449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb49f0eb7b0a48ec8892d6f81ae5e0d0" candidate_uuid: "6be15d44699b45ba8e6dcf690a1280bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dae3adade4a44a6b5efae6081091956"
I20260923 14:45:35.746268 32637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb49f0eb7b0a48ec8892d6f81ae5e0d0" candidate_uuid: "6be15d44699b45ba8e6dcf690a1280bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cef62d399340848985c5cc600c03f9"
I20260923 14:45:35.746400 449 raft_consensus.cc:3037] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:35.746419 32637 raft_consensus.cc:3037] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:35.747339 449 raft_consensus.cc:2454] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be15d44699b45ba8e6dcf690a1280bf in term 1.
I20260923 14:45:35.747576 32637 raft_consensus.cc:2454] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be15d44699b45ba8e6dcf690a1280bf in term 1.
I20260923 14:45:35.747635 32710 leader_election.cc:302] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6be15d44699b45ba8e6dcf690a1280bf, 8dae3adade4a44a6b5efae6081091956; no voters:
I20260923 14:45:35.748020 622 raft_consensus.cc:2781] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:35.748224 622 raft_consensus.cc:686] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 1 LEADER]: Becoming Leader. State: Replica: 6be15d44699b45ba8e6dcf690a1280bf, State: Running, Role: LEADER
I20260923 14:45:35.748366 622 consensus_queue.cc:237] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } }
I20260923 14:45:35.750259 32477 catalog_manager.cc:5495] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194). New cstate: current_term: 1 leader_uuid: "6be15d44699b45ba8e6dcf690a1280bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6be15d44699b45ba8e6dcf690a1280bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20260923 14:45:35.787708 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:35.784730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52372 (local address 127.30.2.194:41029)
0923 14:45:35.785028 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:35.785035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:35.785047 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:45:35.785089 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:35.785091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:35.785134 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:35.785208 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:35.785426 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:35.785959 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:35.786734 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:35.786957 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:35.787039 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:35.787258 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:35.787330 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:35.787384 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:35.787467 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 14:45:35.787509 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":137,"threads_started":1}
I20260923 14:45:35.794173 449 raft_consensus.cc:1261] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Refusing update from remote peer 6be15d44699b45ba8e6dcf690a1280bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:35.794209 32637 raft_consensus.cc:1261] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9 [term 1 FOLLOWER]: Refusing update from remote peer 6be15d44699b45ba8e6dcf690a1280bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:35.794651 635 consensus_queue.cc:1035] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dae3adade4a44a6b5efae6081091956" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:35.794886 622 consensus_queue.cc:1035] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38cef62d399340848985c5cc600c03f9" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 14:45:35.808252 586 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 14:45:37.363914 32477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:37.364074 32477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:37.366505 32477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3f92d71cfc84cf9aeea31bf0090f150: Sending DeleteTablet for 3 replicas of tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0
I20260923 14:45:37.367122 32617 tablet_service.cc:1514] Processing DeleteTablet for tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:37 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:45366
I20260923 14:45:37.367182 32757 tablet_service.cc:1514] Processing DeleteTablet for tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:37 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:52356
I20260923 14:45:37.367189 428 tablet_service.cc:1514] Processing DeleteTablet for tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:37 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42810
I20260923 14:45:37.367518 666 tablet_replica.cc:330] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf: stopping tablet replica
I20260923 14:45:37.367559 665 tablet_replica.cc:330] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 38cef62d399340848985c5cc600c03f9: stopping tablet replica
I20260923 14:45:37.367604 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 32552
I20260923 14:45:37.367651 667 tablet_replica.cc:330] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956: stopping tablet replica
I20260923 14:45:37.368424 667 raft_consensus.cc:2229] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:37.368563 667 raft_consensus.cc:2258] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:37.369335 667 ts_tablet_manager.cc:1905] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:37.369357 666 raft_consensus.cc:2229] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:45:37.369535 666 raft_consensus.cc:2258] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:37.370316 666 ts_tablet_manager.cc:1905] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:37.371913 667 ts_tablet_manager.cc:1918] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 14:45:37.371966 667 log.cc:1196] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/eb49f0eb7b0a48ec8892d6f81ae5e0d0
I20260923 14:45:37.372171 667 ts_tablet_manager.cc:1939] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 8dae3adade4a44a6b5efae6081091956: Deleting consensus metadata
I20260923 14:45:37.373588 666 ts_tablet_manager.cc:1918] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 14:45:37.373658 666 log.cc:1196] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/eb49f0eb7b0a48ec8892d6f81ae5e0d0
I20260923 14:45:37.373914 666 ts_tablet_manager.cc:1939] T eb49f0eb7b0a48ec8892d6f81ae5e0d0 P 6be15d44699b45ba8e6dcf690a1280bf: Deleting consensus metadata
I20260923 14:45:37.374683 32466 catalog_manager.cc:4841] TS 6be15d44699b45ba8e6dcf690a1280bf (127.30.2.194:41029): tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0 (table test-table [id=ce1069abc1144e31b47ab2d7946a16de]) successfully deleted
I20260923 14:45:37.374883 32466 catalog_manager.cc:4841] TS 8dae3adade4a44a6b5efae6081091956 (127.30.2.195:33889): tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0 (table test-table [id=ce1069abc1144e31b47ab2d7946a16de]) successfully deleted
W20260923 14:45:37.377833 32465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:40767 (error 108)
I20260923 14:45:37.378095 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 32692
I20260923 14:45:37.379571 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:37.379066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:40767 (local address 127.0.0.1:45378)
0923 14:45:37.379346 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:37.379438 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
W20260923 14:45:37.380193 32465 catalog_manager.cc:4584] TS 38cef62d399340848985c5cc600c03f9 (127.30.2.193:40767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb49f0eb7b0a48ec8892d6f81ae5e0d0: Network error: Client connection negotiation failed: client connection to 127.30.2.193:40767: connect: Connection refused (error 111)
I20260923 14:45:37.386060 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 364
I20260923 14:45:37.393425 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 32448
2026-09-23T14:45:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6311 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:37 dist-test-slave-wjf2 krb5kdc[672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:37 dist-test-slave-wjf2 krb5kdc[672](info): set up 2 sockets
Sep 23 14:45:37 dist-test-slave-wjf2 krb5kdc[672](info): commencing operation
krb5kdc: starting...
W20260923 14:45:38.441030 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:38 dist-test-slave-wjf2 krb5kdc[672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:45:38.564080 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:40413
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:40413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:38.677210 685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:38.677528 685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:38.677608 685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:38.681680 685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:38.681782 685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:38.681826 685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:38.681890 685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:38.681934 685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:38.687285 685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:40413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:40413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:38.688553 685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:38.689659 685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:38.698606 690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:38.698647 691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:38.698658 693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:38.698846 685 server_base.cc:1034] running on GCE node
I20260923 14:45:38.699481 685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:38.700637 685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:38.701798 685 hybrid_clock.cc:648] HybridClock initialized: now 1790174738701764 us; error 47 us; skew 500 ppm
Sep 23 14:45:38 dist-test-slave-wjf2 krb5kdc[672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174738, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:38.704448 685 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:45:38.705330 685 webserver.cc:460] Webserver started at http://127.30.2.254:38611/ using document root <none> and password file <none>
I20260923 14:45:38.705839 685 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:38.705895 685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:38.706056 685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:38.708084 685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "4a52cfa9f5bb4fd2b0236ae36ea85d95"
format_stamp: "Formatted at 2026-09-23 14:45:38 on dist-test-slave-wjf2"
server_key: "554226f3c8e3af61d9b7286347fa3aa3"
server_key_iv: "511f25ed5c04ff9683ecb2cf537be83a"
server_key_version: "encryptionkey@0"
I20260923 14:45:38.708590 685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "4a52cfa9f5bb4fd2b0236ae36ea85d95"
format_stamp: "Formatted at 2026-09-23 14:45:38 on dist-test-slave-wjf2"
server_key: "554226f3c8e3af61d9b7286347fa3aa3"
server_key_iv: "511f25ed5c04ff9683ecb2cf537be83a"
server_key_version: "encryptionkey@0"
I20260923 14:45:38.712139 685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:45:38.714437 700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:38.715720 685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:38.715903 685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "4a52cfa9f5bb4fd2b0236ae36ea85d95"
format_stamp: "Formatted at 2026-09-23 14:45:38 on dist-test-slave-wjf2"
server_key: "554226f3c8e3af61d9b7286347fa3aa3"
server_key_iv: "511f25ed5c04ff9683ecb2cf537be83a"
server_key_version: "encryptionkey@0"
I20260923 14:45:38.716001 685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:38.750658 685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:38.754384 685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:38.754591 685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:38.762087 685 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:40413
I20260923 14:45:38.762101 752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:40413 every 8 connection(s)
I20260923 14:45:38.763077 685 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:38.765862 753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:38.770117 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 685
I20260923 14:45:38.770234 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:38.770547 30731 external_mini_cluster.cc:1428] Setting key 7f680cd9e2c9854bf39d02496dd01089
I20260923 14:45:38.772424 753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95: Bootstrap starting.
I20260923 14:45:38.775184 753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:38.776101 753 log.cc:824] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:38.778324 753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95: No bootstrap required, opened a new log
Sep 23 14:45:38 dist-test-slave-wjf2 krb5kdc[672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:38.782493 753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a52cfa9f5bb4fd2b0236ae36ea85d95" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40413 } }
I20260923 14:45:38.782768 753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:38.782835 753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a52cfa9f5bb4fd2b0236ae36ea85d95, State: Initialized, Role: FOLLOWER
I20260923 14:45:38.783370 753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a52cfa9f5bb4fd2b0236ae36ea85d95" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40413 } }
I20260923 14:45:38.783543 753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:38.783618 753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:38.783720 753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:38.784886 753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a52cfa9f5bb4fd2b0236ae36ea85d95" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40413 } }
I20260923 14:45:38.785259 753 leader_election.cc:302] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a52cfa9f5bb4fd2b0236ae36ea85d95; no voters:
I20260923 14:45:38.785604 753 leader_election.cc:288] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:38.785686 758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:38.785951 758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [term 1 LEADER]: Becoming Leader. State: Replica: 4a52cfa9f5bb4fd2b0236ae36ea85d95, State: Running, Role: LEADER
I20260923 14:45:38.786283 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:38.772090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.30.2.254:40413)
0923 14:45:38.772506 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:38.772527 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:38.772570 (+ 43us) server_negotiation.cc:407] Connection header received
0923 14:45:38.773386 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:38.773410 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:38.773876 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:38.774236 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:38.774924 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:38.776482 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:38.777524 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:38.778190 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:38.778353 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:38.780959 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:38.780967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:38.780989 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:38.781028 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:38.783039 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:38.783500 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:38.783506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:38.783512 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:38.783669 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:38.783988 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:38.783994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:38.783996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:38.784484 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:38.784704 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:38.784952 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 14:45:38.785254 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260923 14:45:38.786345 758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a52cfa9f5bb4fd2b0236ae36ea85d95" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40413 } }
I20260923 14:45:38.786949 753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:38.788129 760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a52cfa9f5bb4fd2b0236ae36ea85d95. Latest consensus state: current_term: 1 leader_uuid: "4a52cfa9f5bb4fd2b0236ae36ea85d95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a52cfa9f5bb4fd2b0236ae36ea85d95" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40413 } } }
I20260923 14:45:38.788265 760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:38.788612 759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a52cfa9f5bb4fd2b0236ae36ea85d95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a52cfa9f5bb4fd2b0236ae36ea85d95" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40413 } } }
I20260923 14:45:38.788690 759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95 [sys.catalog]: This master's current role is: LEADER
W20260923 14:45:38.791710 773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:45:38.791819 773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:45:38.791910 767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:38.792855 767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:38.798465 767 catalog_manager.cc:1347] Generated new cluster ID: fc257deb83bd4a05b352244995bea1f1
I20260923 14:45:38.798532 767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:38.832836 767 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:38.834246 767 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:38.847213 767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a52cfa9f5bb4fd2b0236ae36ea85d95: Generated new TSK 0
I20260923 14:45:38.848143 767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:45:38.880930 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40413
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:38.995800 779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:38.996043 779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:38.996089 779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:39.000217 779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:39.000276 779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:39.000356 779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:39.005703 779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:39.006789 779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:39.007894 779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:39.018200 785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:39.018249 784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:39.018261 779 server_base.cc:1034] running on GCE node
W20260923 14:45:39.018225 787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:39.018867 779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:39.019546 779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:39.020706 779 hybrid_clock.cc:648] HybridClock initialized: now 1790174739020677 us; error 40 us; skew 500 ppm
Sep 23 14:45:39 dist-test-slave-wjf2 krb5kdc[672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:39.023700 779 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:45:39.024598 779 webserver.cc:460] Webserver started at http://127.30.2.193:44063/ using document root <none> and password file <none>
I20260923 14:45:39.025173 779 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:39.025233 779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:39.025460 779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:39.027550 779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "545756104e8a4bfaaedcf5ff1f999bf8"
format_stamp: "Formatted at 2026-09-23 14:45:39 on dist-test-slave-wjf2"
server_key: "e93d49b2453cf65103d7384ec589957c"
server_key_iv: "180b5e0f1e7cf75963e9d39b36467524"
server_key_version: "encryptionkey@0"
I20260923 14:45:39.028160 779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "545756104e8a4bfaaedcf5ff1f999bf8"
format_stamp: "Formatted at 2026-09-23 14:45:39 on dist-test-slave-wjf2"
server_key: "e93d49b2453cf65103d7384ec589957c"
server_key_iv: "180b5e0f1e7cf75963e9d39b36467524"
server_key_version: "encryptionkey@0"
I20260923 14:45:39.031737 779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:45:39.034193 794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:39.035398 779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:39.035539 779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "545756104e8a4bfaaedcf5ff1f999bf8"
format_stamp: "Formatted at 2026-09-23 14:45:39 on dist-test-slave-wjf2"
server_key: "e93d49b2453cf65103d7384ec589957c"
server_key_iv: "180b5e0f1e7cf75963e9d39b36467524"
server_key_version: "encryptionkey@0"
I20260923 14:45:39.035650 779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:39.063411 779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:39.066550 779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:39.066757 779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:39.067449 779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:39.068285 779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:39.068352 779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:39.068425 779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:39.068475 779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:39.077518 779 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:40409
I20260923 14:45:39.077535 907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:40409 every 8 connection(s)
I20260923 14:45:39.078384 779 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:45:39.087821 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 779
I20260923 14:45:39.087939 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:39.088290 30731 external_mini_cluster.cc:1428] Setting key c31763986f16dc7b29fd1264efa3bf56
Sep 23 14:45:39 dist-test-slave-wjf2 krb5kdc[672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:39.093286 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:39.080367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:46855 (local address 127.30.2.254:40413)
0923 14:45:39.080531 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:39.080536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:39.081382 (+ 846us) server_negotiation.cc:407] Connection header received
0923 14:45:39.082674 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:39.082676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:39.082721 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:39.082794 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:39.083937 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.084459 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.085627 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.085873 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.085973 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:39.089051 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:39.089056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:39.089058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:39.089129 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:39.090924 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:39.091462 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:39.091469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:39.091474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:39.091572 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:39.092049 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:39.092051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:39.092053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:39.092270 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:39.092350 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:39.093023 (+ 673us) server_negotiation.cc:299] Negotiation successful
0923 14:45:39.093162 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 14:45:39.094488 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:39.080735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40413 (local address 127.30.2.193:46855)
0923 14:45:39.081234 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:39.081276 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:39.082481 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:39.082958 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:39.082965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:39.083334 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:39.083816 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:39.083830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.084581 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.084584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:39.085465 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:39.085474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.086038 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.086041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:39.086213 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:39.086983 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:39.087000 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:39.088839 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:39.091048 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:39.091054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:39.091074 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:39.091359 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:39.091700 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:39.091706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:39.091710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:39.091942 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:39.092377 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:39.092384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:39.092734 (+ 350us) client_negotiation.cc:769] Sending connection context
0923 14:45:39.092965 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 14:45:39.093340 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":101,"threads_started":1}
I20260923 14:45:39.095990 908 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40413
I20260923 14:45:39.096304 908 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:39.096938 908 heartbeater.cc:511] Master 127.30.2.254:40413 requested a full tablet report, sending...
I20260923 14:45:39.098623 717 ts_manager.cc:194] Registered new tserver with Master: 545756104e8a4bfaaedcf5ff1f999bf8 (127.30.2.193:40409)
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:45:39.099949 717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:46855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:45:39.119515 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40413
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:39.238301 913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:39.238683 913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:39.238771 913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:39.243088 913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:39.243178 913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:39.243285 913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:39.250525 913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:39.251585 913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:39.252693 913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:39.262612 919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:39.262619 918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:39.262674 921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:39.262786 913 server_base.cc:1034] running on GCE node
I20260923 14:45:39.263324 913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:39.264056 913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:39.265234 913 hybrid_clock.cc:648] HybridClock initialized: now 1790174739265216 us; error 28 us; skew 500 ppm
Sep 23 14:45:39 dist-test-slave-wjf2 krb5kdc[672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:39.268111 913 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:45:39.268986 913 webserver.cc:460] Webserver started at http://127.30.2.194:37005/ using document root <none> and password file <none>
I20260923 14:45:39.269542 913 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:39.269611 913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:39.269795 913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:39.271968 913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "bd7601fb6d654266a8776f66747dbcab"
format_stamp: "Formatted at 2026-09-23 14:45:39 on dist-test-slave-wjf2"
server_key: "f32e1b7a055f88f7b188a7a9b81d9d85"
server_key_iv: "3f4c51adc5c9e1fa8690d7b2ed944e95"
server_key_version: "encryptionkey@0"
I20260923 14:45:39.272609 913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "bd7601fb6d654266a8776f66747dbcab"
format_stamp: "Formatted at 2026-09-23 14:45:39 on dist-test-slave-wjf2"
server_key: "f32e1b7a055f88f7b188a7a9b81d9d85"
server_key_iv: "3f4c51adc5c9e1fa8690d7b2ed944e95"
server_key_version: "encryptionkey@0"
I20260923 14:45:39.276446 913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:45:39.278985 928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:39.280395 913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:45:39.280542 913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "bd7601fb6d654266a8776f66747dbcab"
format_stamp: "Formatted at 2026-09-23 14:45:39 on dist-test-slave-wjf2"
server_key: "f32e1b7a055f88f7b188a7a9b81d9d85"
server_key_iv: "3f4c51adc5c9e1fa8690d7b2ed944e95"
server_key_version: "encryptionkey@0"
I20260923 14:45:39.280666 913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:39.312354 913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:39.315794 913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:39.316068 913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:39.316751 913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:39.317672 913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:39.317749 913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:39.317860 913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:39.317919 913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:39.327828 913 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:35697
I20260923 14:45:39.327881 1041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:35697 every 8 connection(s)
I20260923 14:45:39.328804 913 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:39.336361 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 913
I20260923 14:45:39.336484 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:39.336788 30731 external_mini_cluster.cc:1428] Setting key d90431502f75a2dd9ba28d839237b7af
Sep 23 14:45:39 dist-test-slave-wjf2 krb5kdc[672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:39.343493 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:39.330720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:41655 (local address 127.30.2.254:40413)
0923 14:45:39.330903 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:39.330908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:39.331800 (+ 892us) server_negotiation.cc:407] Connection header received
0923 14:45:39.333351 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:39.333355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:39.333405 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:39.333485 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:39.334710 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.335251 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.336407 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.336771 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.336905 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:39.339472 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:39.339479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:39.339485 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:39.339520 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:39.341111 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:39.341799 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:39.341802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:39.341804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:39.341880 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:39.342308 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:39.342310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:39.342312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:39.342490 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:39.342568 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:39.343259 (+ 691us) server_negotiation.cc:299] Negotiation successful
0923 14:45:39.343372 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 14:45:39.344893 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:39.331145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40413 (local address 127.30.2.194:41655)
0923 14:45:39.331620 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:39.331665 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:39.333138 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:39.333692 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:39.333699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:39.334058 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:39.334544 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:39.334557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.335363 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.335365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:39.336265 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:39.336274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.336976 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.336979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:39.337115 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:39.337730 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:39.337747 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:39.339300 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:39.341242 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:39.341252 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:39.341271 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:39.341682 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:39.341980 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:39.341987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:39.341992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:39.342194 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:39.342613 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:39.342623 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:39.342979 (+ 356us) client_negotiation.cc:769] Sending connection context
0923 14:45:39.343226 (+ 247us) client_negotiation.cc:240] Negotiation successful
0923 14:45:39.343650 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260923 14:45:39.346145 1042 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40413
I20260923 14:45:39.346393 1042 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:39.346938 1042 heartbeater.cc:511] Master 127.30.2.254:40413 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:45:39.348168 717 ts_manager.cc:194] Registered new tserver with Master: bd7601fb6d654266a8776f66747dbcab (127.30.2.194:35697)
I20260923 14:45:39.348788 717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:41655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:45:39.366551 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40413
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:39.474398 1047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:39.474686 1047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:39.474773 1047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:39.478788 1047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:39.478863 1047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:39.478967 1047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:39.484098 1047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:39.485155 1047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:39.486145 1047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:39.495797 1052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:39.495867 1047 server_base.cc:1034] running on GCE node
W20260923 14:45:39.495796 1055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:39.495766 1053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:39.496538 1047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:39.497149 1047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:39.498337 1047 hybrid_clock.cc:648] HybridClock initialized: now 1790174739498320 us; error 27 us; skew 500 ppm
Sep 23 14:45:39 dist-test-slave-wjf2 krb5kdc[672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:39.501256 1047 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:45:39.502184 1047 webserver.cc:460] Webserver started at http://127.30.2.195:38229/ using document root <none> and password file <none>
I20260923 14:45:39.502807 1047 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:39.502914 1047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:39.503185 1047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:39.505210 1047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "4f36173492ed43c193d6bf469d8349e0"
format_stamp: "Formatted at 2026-09-23 14:45:39 on dist-test-slave-wjf2"
server_key: "9372321b56229dfe36728717171df0a7"
server_key_iv: "9219d8f195938988e440a177b234781b"
server_key_version: "encryptionkey@0"
I20260923 14:45:39.505776 1047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "4f36173492ed43c193d6bf469d8349e0"
format_stamp: "Formatted at 2026-09-23 14:45:39 on dist-test-slave-wjf2"
server_key: "9372321b56229dfe36728717171df0a7"
server_key_iv: "9219d8f195938988e440a177b234781b"
server_key_version: "encryptionkey@0"
I20260923 14:45:39.509464 1047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:45:39.512030 1062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:39.513331 1047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:45:39.513484 1047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "4f36173492ed43c193d6bf469d8349e0"
format_stamp: "Formatted at 2026-09-23 14:45:39 on dist-test-slave-wjf2"
server_key: "9372321b56229dfe36728717171df0a7"
server_key_iv: "9219d8f195938988e440a177b234781b"
server_key_version: "encryptionkey@0"
I20260923 14:45:39.513597 1047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:39.528282 1047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:39.531013 1047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:39.531205 1047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:39.531872 1047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:39.532680 1047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:39.532747 1047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:39.532816 1047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:39.532858 1047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:39.542411 1047 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:36973
I20260923 14:45:39.542485 1175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:36973 every 8 connection(s)
I20260923 14:45:39.543362 1047 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:45:39.552322 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 1047
I20260923 14:45:39.552484 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:45:39.552752 30731 external_mini_cluster.cc:1428] Setting key b95818317c08b7d41c58ad3d3d37da8d
Sep 23 14:45:39 dist-test-slave-wjf2 krb5kdc[672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:39.557538 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:39.545311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:35553 (local address 127.30.2.254:40413)
0923 14:45:39.545469 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:39.545474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:39.546342 (+ 868us) server_negotiation.cc:407] Connection header received
0923 14:45:39.547653 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:39.547656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:39.547705 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:39.547822 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:39.548983 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.549508 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.550590 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.550799 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.550874 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:39.553810 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:39.553813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:39.553816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:39.553842 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:39.555672 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:39.556194 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:39.556196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:39.556198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:39.556279 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:39.556579 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:39.556581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:39.556582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:39.556756 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:39.556835 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:39.557323 (+ 488us) server_negotiation.cc:299] Negotiation successful
0923 14:45:39.557421 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:45:39.558317 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:39.545744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40413 (local address 127.30.2.195:35553)
0923 14:45:39.546186 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:39.546228 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:39.547450 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:39.547989 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:39.547996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:39.548374 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:39.548838 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:39.548852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.549611 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.549614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:39.550483 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:39.550492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:39.550919 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:39.550922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:39.551062 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:39.551685 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:39.551701 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:39.553628 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:39.555825 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:39.555830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:39.555844 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:39.556088 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:39.556368 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:39.556370 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:39.556371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:39.556492 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:39.556847 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:39.556852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:39.557059 (+ 207us) client_negotiation.cc:769] Sending connection context
0923 14:45:39.557223 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 14:45:39.557452 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260923 14:45:39.559512 1176 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40413
I20260923 14:45:39.559746 1176 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:39.560246 1176 heartbeater.cc:511] Master 127.30.2.254:40413 requested a full tablet report, sending...
I20260923 14:45:39.561266 717 ts_manager.cc:194] Registered new tserver with Master: 4f36173492ed43c193d6bf469d8349e0 (127.30.2.195:36973)
I20260923 14:45:39.561812 717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:35553
I20260923 14:45:39.566781 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 14:45:39.579035 756 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:45:39.577759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.30.2.254:40413)
0923 14:45:39.577893 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:39.577897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:39.578011 (+ 114us) server_negotiation.cc:407] Connection header received
0923 14:45:39.578157 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:39.578160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:39.578206 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:39.578281 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:39.578936 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33186: BlockingRecv error: recv got EOF from 127.0.0.1:33186 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:45:39.580259 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 779
I20260923 14:45:39.586313 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 913
I20260923 14:45:39.591919 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 1047
I20260923 14:45:39.597127 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 685
2026-09-23T14:45:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2202 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:39 dist-test-slave-wjf2 krb5kdc[1188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:39 dist-test-slave-wjf2 krb5kdc[1188](info): set up 2 sockets
Sep 23 14:45:39 dist-test-slave-wjf2 krb5kdc[1188](info): commencing operation
krb5kdc: starting...
W20260923 14:45:40.629151 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:40 dist-test-slave-wjf2 krb5kdc[1188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:45:40.752925 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44361
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:44361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:40.865288 1201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:40.865528 1201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:40.865567 1201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:40.869524 1201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:40.869575 1201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:40.869598 1201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:40.869616 1201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:40.869629 1201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:40.874756 1201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:44361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:40.875742 1201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:40.876816 1201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:40.885612 1206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:40.885702 1207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:40.885612 1209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:40.885936 1201 server_base.cc:1034] running on GCE node
I20260923 14:45:40.886497 1201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:40.887728 1201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:40.888971 1201 hybrid_clock.cc:648] HybridClock initialized: now 1790174740888936 us; error 47 us; skew 500 ppm
Sep 23 14:45:40 dist-test-slave-wjf2 krb5kdc[1188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174740, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:40.898227 1201 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:45:40.899214 1201 webserver.cc:460] Webserver started at http://127.30.2.254:35327/ using document root <none> and password file <none>
I20260923 14:45:40.899873 1201 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:40.899943 1201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:40.900146 1201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:40.902377 1201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "845fa1616a6b435ba276cd0b2c1350a4"
format_stamp: "Formatted at 2026-09-23 14:45:40 on dist-test-slave-wjf2"
server_key: "18232a995e304428c39a1ff04ba89613"
server_key_iv: "28939a9302e5f81eed401a6f170f92f3"
server_key_version: "encryptionkey@0"
I20260923 14:45:40.903025 1201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "845fa1616a6b435ba276cd0b2c1350a4"
format_stamp: "Formatted at 2026-09-23 14:45:40 on dist-test-slave-wjf2"
server_key: "18232a995e304428c39a1ff04ba89613"
server_key_iv: "28939a9302e5f81eed401a6f170f92f3"
server_key_version: "encryptionkey@0"
I20260923 14:45:40.907248 1201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 14:45:40.909978 1216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:40.911370 1201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:45:40.911520 1201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "845fa1616a6b435ba276cd0b2c1350a4"
format_stamp: "Formatted at 2026-09-23 14:45:40 on dist-test-slave-wjf2"
server_key: "18232a995e304428c39a1ff04ba89613"
server_key_iv: "28939a9302e5f81eed401a6f170f92f3"
server_key_version: "encryptionkey@0"
I20260923 14:45:40.911635 1201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:40.932588 1201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:40.935740 1201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:40.935989 1201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:40.942946 1201 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:44361
I20260923 14:45:40.943228 1268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:44361 every 8 connection(s)
I20260923 14:45:40.943969 1201 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:40.946833 1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:40.948840 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 1201
I20260923 14:45:40.948939 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:40.949380 30731 external_mini_cluster.cc:1428] Setting key 320900b3741a6e02e9b035da6182bc39
I20260923 14:45:40.953181 1269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4: Bootstrap starting.
I20260923 14:45:40.955747 1269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:40.956578 1269 log.cc:824] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:40.958586 1269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4: No bootstrap required, opened a new log
Sep 23 14:45:40 dist-test-slave-wjf2 krb5kdc[1188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:40.962054 1269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fa1616a6b435ba276cd0b2c1350a4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44361 } }
I20260923 14:45:40.962314 1269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:40.962386 1269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845fa1616a6b435ba276cd0b2c1350a4, State: Initialized, Role: FOLLOWER
I20260923 14:45:40.962934 1269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fa1616a6b435ba276cd0b2c1350a4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44361 } }
I20260923 14:45:40.963076 1269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:40.963165 1269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:40.963268 1269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:40.964335 1269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fa1616a6b435ba276cd0b2c1350a4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44361 } }
I20260923 14:45:40.964690 1269 leader_election.cc:302] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 845fa1616a6b435ba276cd0b2c1350a4; no voters:
I20260923 14:45:40.965013 1269 leader_election.cc:288] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:40.965072 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:40.950828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.30.2.254:44361)
0923 14:45:40.951290 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:40.951309 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:40.951352 (+ 43us) server_negotiation.cc:407] Connection header received
0923 14:45:40.952201 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:40.952227 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:40.952640 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:40.952999 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:40.953840 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:40.955330 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:40.956345 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:40.957040 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:40.957263 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:40.959686 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:40.959691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:40.959705 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:40.959732 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:40.961882 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:40.962333 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:40.962339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:40.962344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:40.962592 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:40.962875 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:40.962878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:40.962880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:40.963396 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:40.963623 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:40.963919 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 14:45:40.964200 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":116,"threads_started":1}
I20260923 14:45:40.965099 1274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:40.965451 1274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [term 1 LEADER]: Becoming Leader. State: Replica: 845fa1616a6b435ba276cd0b2c1350a4, State: Running, Role: LEADER
I20260923 14:45:40.965807 1274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fa1616a6b435ba276cd0b2c1350a4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44361 } }
I20260923 14:45:40.966360 1269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:40.967396 1275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "845fa1616a6b435ba276cd0b2c1350a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fa1616a6b435ba276cd0b2c1350a4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44361 } } }
I20260923 14:45:40.967597 1275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:40.967927 1275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 845fa1616a6b435ba276cd0b2c1350a4. Latest consensus state: current_term: 1 leader_uuid: "845fa1616a6b435ba276cd0b2c1350a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fa1616a6b435ba276cd0b2c1350a4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44361 } } }
I20260923 14:45:40.968070 1275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:40.968442 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:45:40.968946 1283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:40.972074 1283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:40.978470 1283 catalog_manager.cc:1347] Generated new cluster ID: 4a24129695c3479f9e016243dd30df34
I20260923 14:45:40.978583 1283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:40.993469 1283 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:40.994593 1283 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:40.994910 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:40.989738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.30.2.254:44361)
0923 14:45:40.989858 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:40.989863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:40.989880 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:45:40.989994 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:40.989998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:40.990045 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:40.990151 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:40.990676 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:40.991625 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:40.992630 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:40.992946 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:40.993058 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:40.994002 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:40.994006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:40.994009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:40.994438 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:40.994466 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:40.994494 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:40.994597 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:40.994728 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 14:45:40.994805 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":47}
I20260923 14:45:41.005199 1283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 845fa1616a6b435ba276cd0b2c1350a4: Generated new TSK 0
I20260923 14:45:41.005904 1283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:45:41.015831 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 1201
2026-09-23T14:45:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1417 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:41 dist-test-slave-wjf2 krb5kdc[1302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:41 dist-test-slave-wjf2 krb5kdc[1302](info): set up 2 sockets
Sep 23 14:45:41 dist-test-slave-wjf2 krb5kdc[1302](info): commencing operation
krb5kdc: starting...
W20260923 14:45:42.081923 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:42 dist-test-slave-wjf2 krb5kdc[1302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:45:42.206818 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44419
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:44419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:42.319499 1315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:42.319753 1315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:42.319861 1315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:42.323731 1315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:42.323845 1315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:42.323911 1315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:42.323957 1315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:42.323992 1315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:42.329133 1315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:44419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:42.330112 1315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:42.331102 1315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:42.339380 1321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:42.339380 1320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:42.339402 1323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:42.339876 1315 server_base.cc:1034] running on GCE node
I20260923 14:45:42.340565 1315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:42.341624 1315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:42.342788 1315 hybrid_clock.cc:648] HybridClock initialized: now 1790174742342774 us; error 24 us; skew 500 ppm
Sep 23 14:45:42 dist-test-slave-wjf2 krb5kdc[1302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174742, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:42.345563 1315 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:45:42.346400 1315 webserver.cc:460] Webserver started at http://127.30.2.254:38693/ using document root <none> and password file <none>
I20260923 14:45:42.346972 1315 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:42.347062 1315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:42.347321 1315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:42.349346 1315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "f0fd1e641be94a869bf737623ce20191"
format_stamp: "Formatted at 2026-09-23 14:45:42 on dist-test-slave-wjf2"
server_key: "e6b606a194a3468629e278154a77b333"
server_key_iv: "823e2eaf79ba60fcba9565e231ecbfe3"
server_key_version: "encryptionkey@0"
I20260923 14:45:42.349895 1315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "f0fd1e641be94a869bf737623ce20191"
format_stamp: "Formatted at 2026-09-23 14:45:42 on dist-test-slave-wjf2"
server_key: "e6b606a194a3468629e278154a77b333"
server_key_iv: "823e2eaf79ba60fcba9565e231ecbfe3"
server_key_version: "encryptionkey@0"
I20260923 14:45:42.353468 1315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:45:42.355770 1330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:42.357020 1315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:42.357177 1315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "f0fd1e641be94a869bf737623ce20191"
format_stamp: "Formatted at 2026-09-23 14:45:42 on dist-test-slave-wjf2"
server_key: "e6b606a194a3468629e278154a77b333"
server_key_iv: "823e2eaf79ba60fcba9565e231ecbfe3"
server_key_version: "encryptionkey@0"
I20260923 14:45:42.357283 1315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:42.414706 1315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:42.417847 1315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:42.417994 1315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:42.424959 1315 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:44419
I20260923 14:45:42.425287 1382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:44419 every 8 connection(s)
I20260923 14:45:42.425930 1315 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:42.428824 1383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:42.432971 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 1315
I20260923 14:45:42.433166 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:42.433501 30731 external_mini_cluster.cc:1428] Setting key cc9c2c8bbe896cac03c8523f605d9919
I20260923 14:45:42.434931 1383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191: Bootstrap starting.
I20260923 14:45:42.437299 1383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:42.438215 1383 log.cc:824] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:42.440268 1383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191: No bootstrap required, opened a new log
Sep 23 14:45:42 dist-test-slave-wjf2 krb5kdc[1302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:42.444715 1383 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd1e641be94a869bf737623ce20191" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44419 } }
I20260923 14:45:42.444940 1383 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:42.444998 1383 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0fd1e641be94a869bf737623ce20191, State: Initialized, Role: FOLLOWER
I20260923 14:45:42.445437 1383 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd1e641be94a869bf737623ce20191" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44419 } }
I20260923 14:45:42.445559 1383 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:42.445708 1383 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:42.445823 1383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:42.446648 1383 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd1e641be94a869bf737623ce20191" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44419 } }
I20260923 14:45:42.446987 1383 leader_election.cc:302] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0fd1e641be94a869bf737623ce20191; no voters:
I20260923 14:45:42.447310 1383 leader_election.cc:288] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:42.447386 1388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:42.447613 1388 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [term 1 LEADER]: Becoming Leader. State: Replica: f0fd1e641be94a869bf737623ce20191, State: Running, Role: LEADER
I20260923 14:45:42.447990 1388 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd1e641be94a869bf737623ce20191" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44419 } }
I20260923 14:45:42.448678 1383 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:42.448930 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:42.434993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.30.2.254:44419)
0923 14:45:42.435506 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:42.435525 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:42.435571 (+ 46us) server_negotiation.cc:407] Connection header received
0923 14:45:42.436422 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:42.436442 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:42.436828 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:42.437170 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:42.438078 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:42.439140 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:42.440286 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:42.440786 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:42.440903 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:42.443623 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:42.443629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:42.443647 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:42.443674 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:42.445554 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:42.446126 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:42.446139 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:42.446145 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:42.446255 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:42.446540 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:42.446545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:42.446546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:42.447035 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:42.447233 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:42.447615 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 14:45:42.447989 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":152,"threads_started":1}
I20260923 14:45:42.449235 1388 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0fd1e641be94a869bf737623ce20191. Latest consensus state: current_term: 1 leader_uuid: "f0fd1e641be94a869bf737623ce20191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd1e641be94a869bf737623ce20191" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44419 } } }
I20260923 14:45:42.449424 1388 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:42.449698 1389 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0fd1e641be94a869bf737623ce20191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd1e641be94a869bf737623ce20191" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44419 } } }
I20260923 14:45:42.449846 1389 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:42.449864 1395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:42.453063 1395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:42.458449 1395 catalog_manager.cc:1347] Generated new cluster ID: c584097bbf3c4364938656d69529d5ef
I20260923 14:45:42.458511 1395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:42.481937 1395 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:42.482901 1395 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:42.494102 1395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0fd1e641be94a869bf737623ce20191: Generated new TSK 0
I20260923 14:45:42.494868 1395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:45:42.524833 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44419
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:42.637547 1409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:42.637794 1409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:42.637835 1409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:42.641793 1409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:42.641865 1409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:42.641920 1409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:42.648078 1409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:42.649122 1409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:42.650274 1409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:42.660169 1417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:42.660194 1414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:42.660357 1409 server_base.cc:1034] running on GCE node
W20260923 14:45:42.660177 1415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:42.660908 1409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:42.661650 1409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:42.662855 1409 hybrid_clock.cc:648] HybridClock initialized: now 1790174742662836 us; error 33 us; skew 500 ppm
Sep 23 14:45:42 dist-test-slave-wjf2 krb5kdc[1302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174742, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:42.665776 1409 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:45:42.666667 1409 webserver.cc:460] Webserver started at http://127.30.2.193:44285/ using document root <none> and password file <none>
I20260923 14:45:42.667275 1409 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:42.667375 1409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:42.667625 1409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:42.669796 1409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "5dbd88e1b02546db9cde3e05b4a70144"
format_stamp: "Formatted at 2026-09-23 14:45:42 on dist-test-slave-wjf2"
server_key: "449d13532223e1bd1298139056feb00b"
server_key_iv: "0b8802adaa75e92ed6a4fdc384f451f9"
server_key_version: "encryptionkey@0"
I20260923 14:45:42.670403 1409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "5dbd88e1b02546db9cde3e05b4a70144"
format_stamp: "Formatted at 2026-09-23 14:45:42 on dist-test-slave-wjf2"
server_key: "449d13532223e1bd1298139056feb00b"
server_key_iv: "0b8802adaa75e92ed6a4fdc384f451f9"
server_key_version: "encryptionkey@0"
I20260923 14:45:42.674391 1409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 14:45:42.676923 1424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:42.678172 1409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:45:42.678319 1409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "5dbd88e1b02546db9cde3e05b4a70144"
format_stamp: "Formatted at 2026-09-23 14:45:42 on dist-test-slave-wjf2"
server_key: "449d13532223e1bd1298139056feb00b"
server_key_iv: "0b8802adaa75e92ed6a4fdc384f451f9"
server_key_version: "encryptionkey@0"
I20260923 14:45:42.678433 1409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:42.690184 1409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:42.693311 1409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:42.693540 1409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:42.694196 1409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:42.695128 1409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:42.695225 1409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:42.695304 1409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:42.695365 1409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:42.705886 1409 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:46821
I20260923 14:45:42.705902 1537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:46821 every 8 connection(s)
I20260923 14:45:42.706866 1409 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:45:42.710608 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 1409
I20260923 14:45:42.710770 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:42.711100 30731 external_mini_cluster.cc:1428] Setting key 6eb739790809cb9738b239ba7cd49a21
Sep 23 14:45:42 dist-test-slave-wjf2 krb5kdc[1302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174742, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:45:42.722957 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:42.708873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:58461 (local address 127.30.2.254:44419)
0923 14:45:42.709059 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:42.709065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:42.709918 (+ 853us) server_negotiation.cc:407] Connection header received
0923 14:45:42.711575 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:42.711579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:42.711646 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:42.711771 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:42.713270 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:42.713802 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:42.715329 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:42.715803 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:42.715933 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:42.718878 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:42.718885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:42.718889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:42.718919 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:42.720729 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:42.721311 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:42.721315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:42.721317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:42.721412 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:42.721755 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:42.721757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:42.721758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:42.721933 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:42.722011 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:42.722728 (+ 717us) server_negotiation.cc:299] Negotiation successful
0923 14:45:42.722832 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
I20260923 14:45:42.724304 1540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:42.709304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:44419 (local address 127.30.2.193:58461)
0923 14:45:42.709762 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:42.709804 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:42.711340 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:42.711999 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:42.712007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:42.712402 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:42.713097 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:42.713122 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:42.713946 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:42.713954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:42.715190 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:42.715202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:42.715954 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:42.715958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:42.716157 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:42.716839 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:42.716901 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:42.718699 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:42.720860 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:42.720866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:42.720878 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:42.721161 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:42.721516 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:42.721519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:42.721520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:42.721665 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:42.722133 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:42.722176 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:42.722483 (+ 307us) client_negotiation.cc:769] Sending connection context
0923 14:45:42.722697 (+ 214us) client_negotiation.cc:240] Negotiation successful
0923 14:45:42.723069 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":133,"threads_started":1}
I20260923 14:45:42.725740 1538 heartbeater.cc:347] Connected to a master server at 127.30.2.254:44419
I20260923 14:45:42.726018 1538 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:42.726645 1538 heartbeater.cc:511] Master 127.30.2.254:44419 requested a full tablet report, sending...
I20260923 14:45:42.728587 1347 ts_manager.cc:194] Registered new tserver with Master: 5dbd88e1b02546db9cde3e05b4a70144 (127.30.2.193:46821)
I20260923 14:45:42.729828 1347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:58461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:45:42.742277 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44419
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:42.851382 1543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:42.851658 1543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:42.851733 1543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:42.855659 1543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:42.855753 1543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:42.855882 1543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:42.861222 1543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:42.862322 1543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:42.863330 1543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:42.872829 1551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:42.872869 1549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:42.873003 1543 server_base.cc:1034] running on GCE node
W20260923 14:45:42.872869 1548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:42.873525 1543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:42.874096 1543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:42.875262 1543 hybrid_clock.cc:648] HybridClock initialized: now 1790174742875237 us; error 33 us; skew 500 ppm
Sep 23 14:45:42 dist-test-slave-wjf2 krb5kdc[1302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174742, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:42.878043 1543 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:45:42.878882 1543 webserver.cc:460] Webserver started at http://127.30.2.194:39805/ using document root <none> and password file <none>
I20260923 14:45:42.879447 1543 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:42.879508 1543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:42.879726 1543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:42.881848 1543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "75a69345e4fd4af694e472b444034caa"
format_stamp: "Formatted at 2026-09-23 14:45:42 on dist-test-slave-wjf2"
server_key: "7495337fbb200b343cb874bf3c887fbe"
server_key_iv: "6f84901abdf8a92db11288fc7253939f"
server_key_version: "encryptionkey@0"
I20260923 14:45:42.882467 1543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "75a69345e4fd4af694e472b444034caa"
format_stamp: "Formatted at 2026-09-23 14:45:42 on dist-test-slave-wjf2"
server_key: "7495337fbb200b343cb874bf3c887fbe"
server_key_iv: "6f84901abdf8a92db11288fc7253939f"
server_key_version: "encryptionkey@0"
I20260923 14:45:42.886207 1543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 14:45:42.888733 1558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:42.889979 1543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:45:42.890072 1543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "75a69345e4fd4af694e472b444034caa"
format_stamp: "Formatted at 2026-09-23 14:45:42 on dist-test-slave-wjf2"
server_key: "7495337fbb200b343cb874bf3c887fbe"
server_key_iv: "6f84901abdf8a92db11288fc7253939f"
server_key_version: "encryptionkey@0"
I20260923 14:45:42.890220 1543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:42.899256 1543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:42.902474 1543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:42.902665 1543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:42.903234 1543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:42.904080 1543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:42.904126 1543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:42.904194 1543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:42.904227 1543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:42.913372 1543 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:36285
I20260923 14:45:42.913434 1671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:36285 every 8 connection(s)
I20260923 14:45:42.914269 1543 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:42.918037 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 1543
I20260923 14:45:42.918139 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:42.918412 30731 external_mini_cluster.cc:1428] Setting key 5ebf1955910a211e16925e9516a25594
Sep 23 14:45:42 dist-test-slave-wjf2 krb5kdc[1302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174742, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:42.928388 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:42.916040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:56637 (local address 127.30.2.254:44419)
0923 14:45:42.916192 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:42.916196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:42.916991 (+ 795us) server_negotiation.cc:407] Connection header received
0923 14:45:42.918455 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:42.918458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:42.918516 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:42.918623 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:42.919873 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:42.920637 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:42.921944 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:42.922181 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:42.922276 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:42.924816 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:42.924825 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:42.924828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:42.924851 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:42.926324 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:42.926843 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:42.926846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:42.926847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:42.926919 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:42.927269 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:42.927272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:42.927274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:42.927452 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:42.927554 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:42.928094 (+ 540us) server_negotiation.cc:299] Negotiation successful
0923 14:45:42.928217 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:45:42.929131 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:42.916417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:44419 (local address 127.30.2.194:56637)
0923 14:45:42.916849 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:42.916889 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:42.918246 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:42.918835 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:42.918845 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:42.919225 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:42.919686 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:42.919700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:42.920769 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:42.920772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:42.921800 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:42.921816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:42.922338 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:42.922343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:42.922554 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:42.923206 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:42.923244 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:42.924653 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:42.926438 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:42.926445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:42.926460 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:42.926716 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:42.927020 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:42.927023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:42.927025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:42.927164 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:42.927564 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:42.927573 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:42.927855 (+ 282us) client_negotiation.cc:769] Sending connection context
0923 14:45:42.928036 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 14:45:42.928337 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260923 14:45:42.930434 1672 heartbeater.cc:347] Connected to a master server at 127.30.2.254:44419
I20260923 14:45:42.930670 1672 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:42.931190 1672 heartbeater.cc:511] Master 127.30.2.254:44419 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
I20260923 14:45:42.932399 1347 ts_manager.cc:194] Registered new tserver with Master: 75a69345e4fd4af694e472b444034caa (127.30.2.194:36285)
I20260923 14:45:42.933178 1347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:56637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:45:42.946722 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44419
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:43.059813 1677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:43.060086 1677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:43.060158 1677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:43.064257 1677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:43.064304 1677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:43.064358 1677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:43.069548 1677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:43.070407 1677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:43.071507 1677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:43.081744 1685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:43.081843 1677 server_base.cc:1034] running on GCE node
W20260923 14:45:43.081763 1682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:43.081755 1683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:43.082518 1677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:43.083189 1677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:43.084378 1677 hybrid_clock.cc:648] HybridClock initialized: now 1790174743084361 us; error 26 us; skew 500 ppm
Sep 23 14:45:43 dist-test-slave-wjf2 krb5kdc[1302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174743, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:43.087148 1677 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:45:43.088124 1677 webserver.cc:460] Webserver started at http://127.30.2.195:33445/ using document root <none> and password file <none>
I20260923 14:45:43.088713 1677 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:43.088800 1677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:43.089038 1677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:43.091038 1677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "fdb14fcee09b4546801eef509d6362ba"
format_stamp: "Formatted at 2026-09-23 14:45:43 on dist-test-slave-wjf2"
server_key: "da63cdd3d0ecf86bc52d6a6abf73b4de"
server_key_iv: "0a2b44456712c9b9c19c7edc0471fc9e"
server_key_version: "encryptionkey@0"
I20260923 14:45:43.091598 1677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "fdb14fcee09b4546801eef509d6362ba"
format_stamp: "Formatted at 2026-09-23 14:45:43 on dist-test-slave-wjf2"
server_key: "da63cdd3d0ecf86bc52d6a6abf73b4de"
server_key_iv: "0a2b44456712c9b9c19c7edc0471fc9e"
server_key_version: "encryptionkey@0"
I20260923 14:45:43.095407 1677 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:45:43.097914 1692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:43.099205 1677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:45:43.099345 1677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "fdb14fcee09b4546801eef509d6362ba"
format_stamp: "Formatted at 2026-09-23 14:45:43 on dist-test-slave-wjf2"
server_key: "da63cdd3d0ecf86bc52d6a6abf73b4de"
server_key_iv: "0a2b44456712c9b9c19c7edc0471fc9e"
server_key_version: "encryptionkey@0"
I20260923 14:45:43.099455 1677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:43.144196 1677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:43.147599 1677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:43.147856 1677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:43.148486 1677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:43.149406 1677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:43.149471 1677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:43.149545 1677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:43.149596 1677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:43.158560 1677 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:46293
I20260923 14:45:43.158602 1805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:46293 every 8 connection(s)
I20260923 14:45:43.159469 1677 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:45:43.163529 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 1677
I20260923 14:45:43.163626 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:45:43.164013 30731 external_mini_cluster.cc:1428] Setting key f049e7f9fac6d241ef07404095599ef4
Sep 23 14:45:43 dist-test-slave-wjf2 krb5kdc[1302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174743, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:43.173105 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:43.161406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:59953 (local address 127.30.2.254:44419)
0923 14:45:43.161559 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:43.161562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:43.162401 (+ 839us) server_negotiation.cc:407] Connection header received
0923 14:45:43.163910 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:43.163914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:43.163976 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:43.164076 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:43.165518 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:43.166129 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:43.167320 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:43.167528 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:43.167617 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:43.169854 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:43.169857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:43.169859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:43.169883 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:43.171350 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:43.171835 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:43.171837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:43.171839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:43.171911 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:43.172222 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:43.172224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:43.172226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:43.172389 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:43.172484 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:43.172898 (+ 414us) server_negotiation.cc:299] Negotiation successful
0923 14:45:43.172986 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:45:43.173880 1808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:43.161788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:44419 (local address 127.30.2.195:59953)
0923 14:45:43.162257 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:43.162298 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:43.163640 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:43.164297 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:43.164322 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:43.164738 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:43.165321 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:43.165362 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:43.166262 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:43.166265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:43.167215 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:43.167225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:43.167670 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:43.167672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:43.167828 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:43.168330 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:43.168346 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:43.169707 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:43.171452 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:43.171456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:43.171467 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:43.171707 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:43.171992 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:43.171995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:43.171996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:43.172120 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:43.172493 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:43.172498 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:43.172706 (+ 208us) client_negotiation.cc:769] Sending connection context
0923 14:45:43.172852 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 14:45:43.173077 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":136,"threads_started":1}
I20260923 14:45:43.174966 1806 heartbeater.cc:347] Connected to a master server at 127.30.2.254:44419
I20260923 14:45:43.175189 1806 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:43.175603 1806 heartbeater.cc:511] Master 127.30.2.254:44419 requested a full tablet report, sending...
I20260923 14:45:43.176651 1347 ts_manager.cc:194] Registered new tserver with Master: fdb14fcee09b4546801eef509d6362ba (127.30.2.195:46293)
I20260923 14:45:43.177414 1347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:59953
I20260923 14:45:43.178049 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 14:45:43 dist-test-slave-wjf2 krb5kdc[1302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174743, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 14:45:43 dist-test-slave-wjf2 krb5kdc[1302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174743, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:43.203207 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:43.195038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.30.2.254:44419)
0923 14:45:43.195178 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:43.195182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:43.195343 (+ 161us) server_negotiation.cc:407] Connection header received
0923 14:45:43.195459 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:43.195461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:43.195505 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:43.195580 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:43.196301 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:43.196815 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:43.197584 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:43.197795 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:43.197868 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:43.200165 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:43.200169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:43.200171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:43.200194 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:43.201592 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:43.201937 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:43.201940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:43.201941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:43.202010 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:43.202312 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:43.202316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:43.202318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:43.202533 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:43.202689 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:43.202917 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 14:45:43.203030 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260923 14:45:43.203589 1345 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49846
I20260923 14:45:43.204506 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 1409
I20260923 14:45:43.211112 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 1543
I20260923 14:45:43.216629 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 1677
I20260923 14:45:43.222335 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 1315
2026-09-23T14:45:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2210 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:43 dist-test-slave-wjf2 krb5kdc[1818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:43 dist-test-slave-wjf2 krb5kdc[1818](info): set up 2 sockets
Sep 23 14:45:43 dist-test-slave-wjf2 krb5kdc[1818](info): commencing operation
krb5kdc: starting...
W20260923 14:45:44.273615 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:44 dist-test-slave-wjf2 krb5kdc[1818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:45:44.395347 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:35673
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:44.509130 1831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:44.509465 1831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:44.509536 1831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:44.513690 1831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:44.513803 1831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:44.513849 1831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:44.513937 1831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:44.513978 1831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:44.519536 1831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:44.520709 1831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:44.521736 1831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:44.530083 1836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:44.530179 1837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:44.530083 1839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:44.530184 1831 server_base.cc:1034] running on GCE node
I20260923 14:45:44.530870 1831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:44.531988 1831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:44.533149 1831 hybrid_clock.cc:648] HybridClock initialized: now 1790174744533117 us; error 42 us; skew 500 ppm
Sep 23 14:45:44 dist-test-slave-wjf2 krb5kdc[1818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174744, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:44.535919 1831 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:45:44.536736 1831 webserver.cc:460] Webserver started at http://127.30.2.254:41725/ using document root <none> and password file <none>
I20260923 14:45:44.537272 1831 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:44.537333 1831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:44.537492 1831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:44.539439 1831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "fdd8cc2fe69e4f108ada71cdbdbe16c0"
format_stamp: "Formatted at 2026-09-23 14:45:44 on dist-test-slave-wjf2"
server_key: "d950b53e3e5b6f37a9db6bd34d7f55f3"
server_key_iv: "827a340d4992628fd3cab884e64c65f5"
server_key_version: "encryptionkey@0"
I20260923 14:45:44.540095 1831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "fdd8cc2fe69e4f108ada71cdbdbe16c0"
format_stamp: "Formatted at 2026-09-23 14:45:44 on dist-test-slave-wjf2"
server_key: "d950b53e3e5b6f37a9db6bd34d7f55f3"
server_key_iv: "827a340d4992628fd3cab884e64c65f5"
server_key_version: "encryptionkey@0"
I20260923 14:45:44.543916 1831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 14:45:44.546355 1846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:44.547626 1831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:44.547745 1831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "fdd8cc2fe69e4f108ada71cdbdbe16c0"
format_stamp: "Formatted at 2026-09-23 14:45:44 on dist-test-slave-wjf2"
server_key: "d950b53e3e5b6f37a9db6bd34d7f55f3"
server_key_iv: "827a340d4992628fd3cab884e64c65f5"
server_key_version: "encryptionkey@0"
I20260923 14:45:44.547919 1831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:44.562345 1831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:44.565343 1831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:44.565558 1831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:44.572700 1831 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:35673
I20260923 14:45:44.572827 1898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:35673 every 8 connection(s)
I20260923 14:45:44.573704 1831 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:44.576696 1899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:44.581449 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 1831
I20260923 14:45:44.581593 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:44.581863 30731 external_mini_cluster.cc:1428] Setting key f37a9f141471451d83f141f967557fd9
I20260923 14:45:44.582808 1899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0: Bootstrap starting.
I20260923 14:45:44.585620 1899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:44.586566 1899 log.cc:824] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:44.588729 1899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0: No bootstrap required, opened a new log
Sep 23 14:45:44 dist-test-slave-wjf2 krb5kdc[1818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:44.592939 1899 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8cc2fe69e4f108ada71cdbdbe16c0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35673 } }
I20260923 14:45:44.593225 1899 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:44.593309 1899 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd8cc2fe69e4f108ada71cdbdbe16c0, State: Initialized, Role: FOLLOWER
I20260923 14:45:44.593850 1899 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8cc2fe69e4f108ada71cdbdbe16c0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35673 } }
I20260923 14:45:44.594019 1899 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:44.594095 1899 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:44.594204 1899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:44.595367 1899 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8cc2fe69e4f108ada71cdbdbe16c0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35673 } }
I20260923 14:45:44.595795 1899 leader_election.cc:302] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdd8cc2fe69e4f108ada71cdbdbe16c0; no voters:
I20260923 14:45:44.596149 1899 leader_election.cc:288] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:44.596374 1904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:44.596652 1904 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [term 1 LEADER]: Becoming Leader. State: Replica: fdd8cc2fe69e4f108ada71cdbdbe16c0, State: Running, Role: LEADER
I20260923 14:45:44.597011 1904 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8cc2fe69e4f108ada71cdbdbe16c0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35673 } }
I20260923 14:45:44.597622 1899 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:44.598110 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:44.583309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.30.2.254:35673)
0923 14:45:44.583742 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:44.583764 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:44.583841 (+ 77us) server_negotiation.cc:407] Connection header received
0923 14:45:44.584761 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:44.584786 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:44.585216 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:44.585598 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:44.586351 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:44.587810 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:44.588678 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:44.589401 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:44.589565 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:44.592112 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:44.592120 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:44.592137 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:44.592171 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:44.594522 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:44.594976 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:44.594985 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:44.594995 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:44.595161 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:44.595451 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:44.595455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:44.595457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:44.596014 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:44.596256 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:44.596612 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 14:45:44.597131 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":111,"threads_started":1}
I20260923 14:45:44.598132 1904 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdd8cc2fe69e4f108ada71cdbdbe16c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8cc2fe69e4f108ada71cdbdbe16c0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35673 } } }
I20260923 14:45:44.598582 1905 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdd8cc2fe69e4f108ada71cdbdbe16c0. Latest consensus state: current_term: 1 leader_uuid: "fdd8cc2fe69e4f108ada71cdbdbe16c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8cc2fe69e4f108ada71cdbdbe16c0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35673 } } }
I20260923 14:45:44.598719 1905 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:44.599170 1913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:44.600028 1904 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:44.602573 1913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:44.608000 1913 catalog_manager.cc:1347] Generated new cluster ID: 5214f5affdde4f709d02e8cc32477f02
I20260923 14:45:44.608064 1913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:44.625780 1913 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:44.627126 1913 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:44.635679 1913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdd8cc2fe69e4f108ada71cdbdbe16c0: Generated new TSK 0
I20260923 14:45:44.636546 1913 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:45:44.673045 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35673
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:44.789070 1925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:44.789345 1925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:44.789391 1925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:44.793431 1925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:44.793497 1925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:44.793558 1925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:44.799140 1925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:44.800364 1925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:44.801380 1925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:44.811275 1933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:44.811299 1931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:44.811407 1925 server_base.cc:1034] running on GCE node
W20260923 14:45:44.811275 1930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:44.812077 1925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:44.812747 1925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:44.813951 1925 hybrid_clock.cc:648] HybridClock initialized: now 1790174744813934 us; error 26 us; skew 500 ppm
Sep 23 14:45:44 dist-test-slave-wjf2 krb5kdc[1818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174744, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:44.817081 1925 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:45:44.818017 1925 webserver.cc:460] Webserver started at http://127.30.2.193:35241/ using document root <none> and password file <none>
I20260923 14:45:44.818637 1925 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:44.818735 1925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:44.818987 1925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:44.821213 1925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "6de5271dc9a54342971bf2f72af82062"
format_stamp: "Formatted at 2026-09-23 14:45:44 on dist-test-slave-wjf2"
server_key: "4e92337d88b8006d12fcd28b8ab49e04"
server_key_iv: "d55336599c1579874f9c6b7d65f4fae6"
server_key_version: "encryptionkey@0"
I20260923 14:45:44.821852 1925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "6de5271dc9a54342971bf2f72af82062"
format_stamp: "Formatted at 2026-09-23 14:45:44 on dist-test-slave-wjf2"
server_key: "4e92337d88b8006d12fcd28b8ab49e04"
server_key_iv: "d55336599c1579874f9c6b7d65f4fae6"
server_key_version: "encryptionkey@0"
I20260923 14:45:44.825906 1925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:45:44.828756 1940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:44.830173 1925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:45:44.830335 1925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "6de5271dc9a54342971bf2f72af82062"
format_stamp: "Formatted at 2026-09-23 14:45:44 on dist-test-slave-wjf2"
server_key: "4e92337d88b8006d12fcd28b8ab49e04"
server_key_iv: "d55336599c1579874f9c6b7d65f4fae6"
server_key_version: "encryptionkey@0"
I20260923 14:45:44.830451 1925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:44.854493 1925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:44.857919 1925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:44.858153 1925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:44.858767 1925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:44.859659 1925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:44.859730 1925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:44.859822 1925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:44.859871 1925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:44.868920 1925 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:35963
I20260923 14:45:44.868949 2053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:35963 every 8 connection(s)
I20260923 14:45:44.869865 1925 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
Sep 23 14:45:44 dist-test-slave-wjf2 krb5kdc[1818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174744, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:44.879563 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 1925
I20260923 14:45:44.879694 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:44.880007 30731 external_mini_cluster.cc:1428] Setting key 64b81957a2922a4738d6f8a1a09eb42e
I20260923 14:45:44.883854 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:44.871619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:47925 (local address 127.30.2.254:35673)
0923 14:45:44.871818 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:44.871824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:44.872705 (+ 881us) server_negotiation.cc:407] Connection header received
0923 14:45:44.874141 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:44.874145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:44.874206 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:44.874311 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:44.875442 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:44.876039 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:44.877173 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:44.877401 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:44.877515 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:44.879890 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:44.879893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:44.879895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:44.879919 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:44.881414 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:44.882031 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:44.882033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:44.882034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:44.882111 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:44.882507 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:44.882509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:44.882511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:44.882688 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:44.882765 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:44.883439 (+ 674us) server_negotiation.cc:299] Negotiation successful
0923 14:45:44.883585 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 14:45:44.884601 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:44.872070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35673 (local address 127.30.2.193:47925)
0923 14:45:44.872550 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:44.872590 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:44.873915 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:44.874491 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:44.874499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:44.874838 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:44.875291 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:44.875304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:44.876177 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:44.876180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:44.877043 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:44.877052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:44.877551 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:44.877555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:44.877726 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:44.878241 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:44.878257 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:44.879703 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:44.881545 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:44.881551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:44.881564 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:44.881934 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:44.882207 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:44.882213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:44.882218 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:44.882410 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:44.882792 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:44.882802 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:44.883122 (+ 320us) client_negotiation.cc:769] Sending connection context
0923 14:45:44.883374 (+ 252us) client_negotiation.cc:240] Negotiation successful
0923 14:45:44.883644 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":101,"threads_started":1}
I20260923 14:45:44.885852 2054 heartbeater.cc:347] Connected to a master server at 127.30.2.254:35673
I20260923 14:45:44.886081 2054 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:44.886569 2054 heartbeater.cc:511] Master 127.30.2.254:35673 requested a full tablet report, sending...
I20260923 14:45:44.888240 1863 ts_manager.cc:194] Registered new tserver with Master: 6de5271dc9a54342971bf2f72af82062 (127.30.2.193:35963)
I20260923 14:45:44.889375 1863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:47925
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:45:44.907392 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35673
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:45.018635 2059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:45.018882 2059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:45.018923 2059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:45.022915 2059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:45.022966 2059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:45.023022 2059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:45.028247 2059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:45.029291 2059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:45.030280 2059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:45.040037 2064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:45.039975 2067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:45.040042 2065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:45.039985 2059 server_base.cc:1034] running on GCE node
I20260923 14:45:45.040551 2059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:45.041122 2059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:45.042292 2059 hybrid_clock.cc:648] HybridClock initialized: now 1790174745042261 us; error 39 us; skew 500 ppm
Sep 23 14:45:45 dist-test-slave-wjf2 krb5kdc[1818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174745, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:45.045142 2059 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:45:45.046061 2059 webserver.cc:460] Webserver started at http://127.30.2.194:39393/ using document root <none> and password file <none>
I20260923 14:45:45.046622 2059 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:45.046708 2059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:45.046984 2059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:45.049023 2059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "21b66be1b85d401fa9c86cdf82cd185d"
format_stamp: "Formatted at 2026-09-23 14:45:45 on dist-test-slave-wjf2"
server_key: "23583093847a78b945ff3d4cf3f6d6da"
server_key_iv: "530378cfb797279b4c6a1c8847498438"
server_key_version: "encryptionkey@0"
I20260923 14:45:45.049602 2059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "21b66be1b85d401fa9c86cdf82cd185d"
format_stamp: "Formatted at 2026-09-23 14:45:45 on dist-test-slave-wjf2"
server_key: "23583093847a78b945ff3d4cf3f6d6da"
server_key_iv: "530378cfb797279b4c6a1c8847498438"
server_key_version: "encryptionkey@0"
I20260923 14:45:45.053496 2059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:45:45.055931 2074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:45.057138 2059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:45.057276 2059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "21b66be1b85d401fa9c86cdf82cd185d"
format_stamp: "Formatted at 2026-09-23 14:45:45 on dist-test-slave-wjf2"
server_key: "23583093847a78b945ff3d4cf3f6d6da"
server_key_iv: "530378cfb797279b4c6a1c8847498438"
server_key_version: "encryptionkey@0"
I20260923 14:45:45.057384 2059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:45.076613 2059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:45.079308 2059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:45.079507 2059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:45.080116 2059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:45.080924 2059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:45.080992 2059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:45.081061 2059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:45.081122 2059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:45.090706 2059 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:38585
I20260923 14:45:45.090734 2187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:38585 every 8 connection(s)
I20260923 14:45:45.091521 2059 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:45.093837 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 2059
I20260923 14:45:45.093967 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:45.094313 30731 external_mini_cluster.cc:1428] Setting key 09721ab9ae5052936fd51766d9dcfcf0
Sep 23 14:45:45 dist-test-slave-wjf2 krb5kdc[1818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174745, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:45:45.106395 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:45.093191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:36799 (local address 127.30.2.254:35673)
0923 14:45:45.093319 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:45.093322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:45.094290 (+ 968us) server_negotiation.cc:407] Connection header received
0923 14:45:45.095945 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:45.095951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:45.096009 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:45.096078 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:45.097633 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.098490 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.099813 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.100033 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.100172 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:45.102644 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:45.102647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:45.102649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:45.102671 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:45.104362 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:45.104883 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:45.104888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:45.104890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:45.104972 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:45.105339 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:45.105344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:45.105346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:45.105530 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:45.105661 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:45.106118 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 14:45:45.106234 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
I20260923 14:45:45.107398 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:45.093582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35673 (local address 127.30.2.194:36799)
0923 14:45:45.094086 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:45.094178 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:45.095690 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:45.096280 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:45.096290 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:45.096783 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:45.097465 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:45.097487 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.098652 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.098655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:45.099573 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:45.099602 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.100227 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.100232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:45.100458 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:45.101059 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:45.101077 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:45.102466 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:45.104505 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:45.104511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:45.104524 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:45.104774 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:45.105064 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:45.105068 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:45.105070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:45.105211 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:45.105658 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:45.105664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:45.105889 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 14:45:45.106074 (+ 185us) client_negotiation.cc:240] Negotiation successful
0923 14:45:45.106455 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":104,"threads_started":1}
I20260923 14:45:45.108763 2188 heartbeater.cc:347] Connected to a master server at 127.30.2.254:35673
I20260923 14:45:45.108996 2188 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:45.109494 2188 heartbeater.cc:511] Master 127.30.2.254:35673 requested a full tablet report, sending...
I20260923 14:45:45.110662 1863 ts_manager.cc:194] Registered new tserver with Master: 21b66be1b85d401fa9c86cdf82cd185d (127.30.2.194:38585)
I20260923 14:45:45.111583 1863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:36799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:45:45.122727 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35673
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:45.232366 2193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:45.232590 2193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:45.232632 2193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:45.236765 2193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:45.236815 2193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:45.236889 2193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:45.242543 2193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:45.243872 2193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:45.244931 2193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:45.255353 2199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:45.255362 2198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:45.255360 2201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:45.255517 2193 server_base.cc:1034] running on GCE node
I20260923 14:45:45.256063 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:45.256613 2193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:45.257777 2193 hybrid_clock.cc:648] HybridClock initialized: now 1790174745257748 us; error 43 us; skew 500 ppm
Sep 23 14:45:45 dist-test-slave-wjf2 krb5kdc[1818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174745, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:45.260579 2193 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:45:45.261459 2193 webserver.cc:460] Webserver started at http://127.30.2.195:34995/ using document root <none> and password file <none>
I20260923 14:45:45.261999 2193 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:45.262084 2193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:45.262351 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:45.264443 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "111d4844834c406eb37709228ececa4b"
format_stamp: "Formatted at 2026-09-23 14:45:45 on dist-test-slave-wjf2"
server_key: "f9bd30bbe468f3e857e7d3e39df1ccef"
server_key_iv: "006ea75f12ec26688787b854138cff02"
server_key_version: "encryptionkey@0"
I20260923 14:45:45.265012 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "111d4844834c406eb37709228ececa4b"
format_stamp: "Formatted at 2026-09-23 14:45:45 on dist-test-slave-wjf2"
server_key: "f9bd30bbe468f3e857e7d3e39df1ccef"
server_key_iv: "006ea75f12ec26688787b854138cff02"
server_key_version: "encryptionkey@0"
I20260923 14:45:45.268661 2193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:45:45.270951 2208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:45.272212 2193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:45.272341 2193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "111d4844834c406eb37709228ececa4b"
format_stamp: "Formatted at 2026-09-23 14:45:45 on dist-test-slave-wjf2"
server_key: "f9bd30bbe468f3e857e7d3e39df1ccef"
server_key_iv: "006ea75f12ec26688787b854138cff02"
server_key_version: "encryptionkey@0"
I20260923 14:45:45.272442 2193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:45.293496 2193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:45.296731 2193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:45.296923 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:45.297514 2193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:45.298417 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:45.298497 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:45.298557 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:45.298669 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:45.308131 2193 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:43195
I20260923 14:45:45.308218 2321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:43195 every 8 connection(s)
I20260923 14:45:45.309123 2193 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
Sep 23 14:45:45 dist-test-slave-wjf2 krb5kdc[1818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174745, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:45.318949 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 2193
I20260923 14:45:45.319070 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:45:45.319341 30731 external_mini_cluster.cc:1428] Setting key d3971a91ce42d9c27dcdf9c9b7dbe6c5
I20260923 14:45:45.323117 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:45.311012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:41499 (local address 127.30.2.254:35673)
0923 14:45:45.311220 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:45.311224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:45.312068 (+ 844us) server_negotiation.cc:407] Connection header received
0923 14:45:45.313502 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:45.313505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:45.313564 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:45.313668 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:45.314838 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.315403 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.316620 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.316834 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.316920 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:45.319555 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:45.319559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:45.319562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:45.319588 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:45.321210 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:45.321818 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:45.321822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:45.321824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:45.321912 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:45.322207 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:45.322209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:45.322210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:45.322378 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:45.322452 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:45.322914 (+ 462us) server_negotiation.cc:299] Negotiation successful
0923 14:45:45.323003 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260923 14:45:45.323992 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:45.311424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35673 (local address 127.30.2.195:41499)
0923 14:45:45.311889 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:45.311951 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:45.313278 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:45.313862 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:45.313869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:45.314222 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:45.314696 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:45.314710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.315559 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.315562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:45.316498 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:45.316508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.316972 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.316975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:45.317117 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:45.317737 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:45.317753 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:45.319368 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:45.321321 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:45.321327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:45.321342 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:45.321720 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:45.321993 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:45.321995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:45.321997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:45.322128 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:45.322467 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:45.322472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:45.322713 (+ 241us) client_negotiation.cc:769] Sending connection context
0923 14:45:45.322866 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 14:45:45.323116 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260923 14:45:45.325065 2322 heartbeater.cc:347] Connected to a master server at 127.30.2.254:35673
I20260923 14:45:45.325302 2322 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:45.325775 2322 heartbeater.cc:511] Master 127.30.2.254:35673 requested a full tablet report, sending...
I20260923 14:45:45.326812 1863 ts_manager.cc:194] Registered new tserver with Master: 111d4844834c406eb37709228ececa4b (127.30.2.195:43195)
I20260923 14:45:45.327466 1863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:41499
I20260923 14:45:45.333461 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 14:45:45 dist-test-slave-wjf2 krb5kdc[1818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 14:45:45 dist-test-slave-wjf2 krb5kdc[1818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:45:45.359381 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:45.349461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.30.2.193:35963)
0923 14:45:45.349774 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:45.349788 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:45.349818 (+ 30us) server_negotiation.cc:407] Connection header received
0923 14:45:45.349866 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:45.349872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:45.350054 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:45.350173 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:45.350985 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.351916 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.352934 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.353308 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.353399 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:45.355862 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:45.355871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:45.355884 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:45.355908 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:45.357567 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:45.358015 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:45.358021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:45.358022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:45.358099 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:45.358372 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:45.358376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:45.358378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:45.358658 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:45.358898 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:45.359057 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 14:45:45.359209 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":112,"threads_started":1}
Sep 23 14:45:45 dist-test-slave-wjf2 krb5kdc[1818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:45.370129 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:45.362339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.30.2.254:35673)
0923 14:45:45.362480 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:45.362485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:45.362560 (+ 75us) server_negotiation.cc:407] Connection header received
0923 14:45:45.362725 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:45.362728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:45.362768 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:45.362847 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:45.363397 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.363978 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.364766 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:45.364995 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:45.365127 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:45.367229 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:45.367233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:45.367235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:45.367266 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:45.368513 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:45.368917 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:45.368921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:45.368923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:45.369000 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:45.369273 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:45.369276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:45.369278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:45.369482 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:45.369622 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:45.369856 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 14:45:45.369969 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 14:45:45.370536 1863 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36476
I20260923 14:45:45.371177 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 1925
I20260923 14:45:45.377296 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 2059
I20260923 14:45:45.383343 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 2193
I20260923 14:45:45.388777 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 1831
2026-09-23T14:45:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2163 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:45 dist-test-slave-wjf2 krb5kdc[2334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:45 dist-test-slave-wjf2 krb5kdc[2334](info): set up 2 sockets
Sep 23 14:45:45 dist-test-slave-wjf2 krb5kdc[2334](info): commencing operation
krb5kdc: starting...
W20260923 14:45:46.433298 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:46 dist-test-slave-wjf2 krb5kdc[2334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:45:46.571311 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:43157
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:46.683351 2347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:46.683616 2347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:46.683658 2347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:46.688103 2347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:46.688171 2347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:46.688191 2347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:46.688241 2347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:46.688273 2347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:46.693599 2347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:43157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:46.694669 2347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:46.695717 2347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:46.704174 2352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:46.704144 2353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:46.704165 2347 server_base.cc:1034] running on GCE node
W20260923 14:45:46.704139 2355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:46.704939 2347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:46.706067 2347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:46.707266 2347 hybrid_clock.cc:648] HybridClock initialized: now 1790174746707229 us; error 48 us; skew 500 ppm
Sep 23 14:45:46 dist-test-slave-wjf2 krb5kdc[2334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174746, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:46.710253 2347 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:45:46.710978 2347 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:46.711074 2347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:46.711380 2347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:46.713510 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "a77c50035b474cbe940e276bda94853b"
format_stamp: "Formatted at 2026-09-23 14:45:46 on dist-test-slave-wjf2"
server_key: "4abcb45696d2ed04e0644e7fb0ad6f15"
server_key_iv: "8be0c0581a80c6466525462793e6cdf8"
server_key_version: "encryptionkey@0"
I20260923 14:45:46.714083 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "a77c50035b474cbe940e276bda94853b"
format_stamp: "Formatted at 2026-09-23 14:45:46 on dist-test-slave-wjf2"
server_key: "4abcb45696d2ed04e0644e7fb0ad6f15"
server_key_iv: "8be0c0581a80c6466525462793e6cdf8"
server_key_version: "encryptionkey@0"
I20260923 14:45:46.717785 2347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 14:45:46.720238 2361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:46.721555 2347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:46.721719 2347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "a77c50035b474cbe940e276bda94853b"
format_stamp: "Formatted at 2026-09-23 14:45:46 on dist-test-slave-wjf2"
server_key: "4abcb45696d2ed04e0644e7fb0ad6f15"
server_key_iv: "8be0c0581a80c6466525462793e6cdf8"
server_key_version: "encryptionkey@0"
I20260923 14:45:46.721836 2347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:46.736572 2347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:46.739874 2347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:46.740103 2347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:46.747900 2347 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:43157
I20260923 14:45:46.748181 2413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:43157 every 8 connection(s)
I20260923 14:45:46.748644 2347 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:46.751487 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:46.757149 2414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b: Bootstrap starting.
I20260923 14:45:46.757412 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 2347
I20260923 14:45:46.757534 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:46.757819 30731 external_mini_cluster.cc:1428] Setting key 60969e7cbcf8c72eca4e64559a87453f
I20260923 14:45:46.759951 2414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:46.760733 2414 log.cc:824] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:46.762626 2414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b: No bootstrap required, opened a new log
I20260923 14:45:46.766022 2414 raft_consensus.cc:348] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c50035b474cbe940e276bda94853b" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43157 } }
I20260923 14:45:46.766260 2414 raft_consensus.cc:374] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:46.766338 2414 raft_consensus.cc:729] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77c50035b474cbe940e276bda94853b, State: Initialized, Role: FOLLOWER
I20260923 14:45:46.766809 2414 consensus_queue.cc:260] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c50035b474cbe940e276bda94853b" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43157 } }
I20260923 14:45:46.766995 2414 raft_consensus.cc:388] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 23 14:45:46 dist-test-slave-wjf2 krb5kdc[2334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:46.767089 2414 raft_consensus.cc:482] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:46.767198 2414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:46.768286 2414 raft_consensus.cc:504] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c50035b474cbe940e276bda94853b" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43157 } }
I20260923 14:45:46.768635 2414 leader_election.cc:302] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a77c50035b474cbe940e276bda94853b; no voters:
I20260923 14:45:46.768972 2414 leader_election.cc:288] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:46.769093 2419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:46.769330 2419 raft_consensus.cc:686] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [term 1 LEADER]: Becoming Leader. State: Replica: a77c50035b474cbe940e276bda94853b, State: Running, Role: LEADER
I20260923 14:45:46.769732 2419 consensus_queue.cc:237] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c50035b474cbe940e276bda94853b" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43157 } }
I20260923 14:45:46.770184 2414 sys_catalog.cc:564] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:46.771368 2421 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a77c50035b474cbe940e276bda94853b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c50035b474cbe940e276bda94853b" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43157 } } }
I20260923 14:45:46.771498 2421 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:46.771757 2421 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a77c50035b474cbe940e276bda94853b. Latest consensus state: current_term: 1 leader_uuid: "a77c50035b474cbe940e276bda94853b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77c50035b474cbe940e276bda94853b" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43157 } } }
I20260923 14:45:46.771874 2421 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:46.773816 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:46.759195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.30.2.254:43157)
0923 14:45:46.759637 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:46.759654 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:46.759692 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:45:46.760509 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:46.760530 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:46.760958 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:46.761329 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:46.762211 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:46.763256 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:46.764244 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:46.764755 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:46.764946 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:46.767832 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:46.767840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:46.767864 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:46.767893 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:46.770142 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:46.770667 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:46.770675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:46.770683 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:46.770833 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:46.771296 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:46.771301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:46.771303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:46.771994 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:46.772278 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:46.772550 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 14:45:46.772856 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":145,"threads_started":1}
W20260923 14:45:46.775059 2434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:45:46.775142 2434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:45:46.775202 2428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:46.776070 2428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:46.781769 2428 catalog_manager.cc:1347] Generated new cluster ID: f8849eda8d844ec18deb2f2ebe5cc3f7
I20260923 14:45:46.781836 2428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:46.796990 2428 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:46.797927 2428 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:46.802881 2428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b: Generated new TSK 0
I20260923 14:45:46.803691 2428 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:45:46.839613 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43157
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:46.951138 2440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:46.951350 2440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:46.951390 2440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:46.955386 2440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:46.955433 2440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:46.955523 2440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:46.961042 2440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:46.961987 2440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:46.962967 2440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:46.972453 2445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:46.972453 2446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:46.972630 2440 server_base.cc:1034] running on GCE node
W20260923 14:45:46.972500 2448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:46.973107 2440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:46.973757 2440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:46.974918 2440 hybrid_clock.cc:648] HybridClock initialized: now 1790174746974905 us; error 25 us; skew 500 ppm
Sep 23 14:45:46 dist-test-slave-wjf2 krb5kdc[2334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174746, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:46.977655 2440 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:45:46.978309 2440 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:46.978367 2440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:46.978540 2440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:46.980537 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "718e673f2812452988d3fa92b19aad94"
format_stamp: "Formatted at 2026-09-23 14:45:46 on dist-test-slave-wjf2"
server_key: "072b9c0281fbec3cc481a7f83b042a00"
server_key_iv: "2628efc19304ab3d6e3c285e8343f4cc"
server_key_version: "encryptionkey@0"
I20260923 14:45:46.981035 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "718e673f2812452988d3fa92b19aad94"
format_stamp: "Formatted at 2026-09-23 14:45:46 on dist-test-slave-wjf2"
server_key: "072b9c0281fbec3cc481a7f83b042a00"
server_key_iv: "2628efc19304ab3d6e3c285e8343f4cc"
server_key_version: "encryptionkey@0"
I20260923 14:45:46.984560 2440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:45:46.986783 2454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:46.988096 2440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:46.988217 2440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "718e673f2812452988d3fa92b19aad94"
format_stamp: "Formatted at 2026-09-23 14:45:46 on dist-test-slave-wjf2"
server_key: "072b9c0281fbec3cc481a7f83b042a00"
server_key_iv: "2628efc19304ab3d6e3c285e8343f4cc"
server_key_version: "encryptionkey@0"
I20260923 14:45:46.988320 2440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:47.008344 2440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:47.011518 2440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:47.011751 2440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:47.012416 2440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:47.013284 2440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:47.013336 2440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:47.013408 2440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:47.013444 2440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:47.022998 2440 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:43761
I20260923 14:45:47.023010 2567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:43761 every 8 connection(s)
I20260923 14:45:47.023620 2440 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:45:47.025815 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 2440
I20260923 14:45:47.025952 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:47.026288 30731 external_mini_cluster.cc:1428] Setting key 2d01b628abd1c616eeab8dd2112e002a
Sep 23 14:45:47 dist-test-slave-wjf2 krb5kdc[2334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174746, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
I20260923 14:45:47.039487 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.025693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:55103 (local address 127.30.2.254:43157)
0923 14:45:47.025853 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:47.025859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:47.026887 (+ 1028us) server_negotiation.cc:407] Connection header received
0923 14:45:47.028537 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:47.028551 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:47.028617 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:47.028724 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:47.030224 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.030765 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.032632 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.032874 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.032970 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.035774 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:47.035816 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:47.035821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:47.035854 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:47.037424 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:47.037966 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:47.037970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:47.037972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:47.038060 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:47.038407 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:47.038414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:47.038417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:47.038599 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:47.038715 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:47.039185 (+ 470us) server_negotiation.cc:299] Negotiation successful
0923 14:45:47.039310 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 14:45:47.040333 2570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.026174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:43157 (local address 127.30.2.193:55103)
0923 14:45:47.026722 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:47.026782 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:47.028299 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:47.028906 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:47.028913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:47.029305 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:47.030017 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.030037 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.030896 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.030903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.032500 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.032513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.033048 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.033055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.033308 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.033950 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:47.033977 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:47.035602 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:47.037587 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:47.037592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:47.037606 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:47.037859 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:47.038169 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:47.038173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:47.038175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:47.038311 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:47.038713 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:47.038720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:47.038955 (+ 235us) client_negotiation.cc:769] Sending connection context
0923 14:45:47.039150 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 14:45:47.039452 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":144,"threads_started":1}
I20260923 14:45:47.041702 2568 heartbeater.cc:347] Connected to a master server at 127.30.2.254:43157
I20260923 14:45:47.041945 2568 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:47.042382 2568 heartbeater.cc:511] Master 127.30.2.254:43157 requested a full tablet report, sending...
I20260923 14:45:47.043987 2378 ts_manager.cc:194] Registered new tserver with Master: 718e673f2812452988d3fa92b19aad94 (127.30.2.193:43761)
I20260923 14:45:47.045069 2378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:55103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:45:47.055491 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43157
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:47.164775 2573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:47.165057 2573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:47.165122 2573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:47.169062 2573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:47.169147 2573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:47.169237 2573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:47.174724 2573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:47.175769 2573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:47.176811 2573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:47.186201 2578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:47.186225 2579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:47.186225 2581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:47.186546 2573 server_base.cc:1034] running on GCE node
I20260923 14:45:47.186993 2573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:47.187614 2573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:47.188809 2573 hybrid_clock.cc:648] HybridClock initialized: now 1790174747188797 us; error 32 us; skew 500 ppm
Sep 23 14:45:47 dist-test-slave-wjf2 krb5kdc[2334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174747, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:47.191829 2573 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:45:47.192729 2573 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:47.192827 2573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:47.193049 2573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:47.195070 2573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "881e1e449a474a82bd5cddd9c34dddaf"
format_stamp: "Formatted at 2026-09-23 14:45:47 on dist-test-slave-wjf2"
server_key: "a895e64d3ecfbffccdea5ab311816eff"
server_key_iv: "6c7bd18997c2753bc6a12d843627a461"
server_key_version: "encryptionkey@0"
I20260923 14:45:47.195659 2573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "881e1e449a474a82bd5cddd9c34dddaf"
format_stamp: "Formatted at 2026-09-23 14:45:47 on dist-test-slave-wjf2"
server_key: "a895e64d3ecfbffccdea5ab311816eff"
server_key_iv: "6c7bd18997c2753bc6a12d843627a461"
server_key_version: "encryptionkey@0"
I20260923 14:45:47.199390 2573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:45:47.201944 2587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:47.203181 2573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:45:47.203274 2573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "881e1e449a474a82bd5cddd9c34dddaf"
format_stamp: "Formatted at 2026-09-23 14:45:47 on dist-test-slave-wjf2"
server_key: "a895e64d3ecfbffccdea5ab311816eff"
server_key_iv: "6c7bd18997c2753bc6a12d843627a461"
server_key_version: "encryptionkey@0"
I20260923 14:45:47.203357 2573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:47.216961 2573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:47.220402 2573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:47.220547 2573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:47.221052 2573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:47.221992 2573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:47.222039 2573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:47.222080 2573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:47.222096 2573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:47.231245 2573 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:36313
I20260923 14:45:47.231302 2700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:36313 every 8 connection(s)
I20260923 14:45:47.232000 2573 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:47.241987 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 2573
I20260923 14:45:47.242136 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:47.242515 30731 external_mini_cluster.cc:1428] Setting key 82bfcc6714e595d6e7c070993bab44d5
Sep 23 14:45:47 dist-test-slave-wjf2 krb5kdc[2334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174747, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:47.249665 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.233804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:46995 (local address 127.30.2.254:43157)
0923 14:45:47.233938 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:47.233941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:47.234779 (+ 838us) server_negotiation.cc:407] Connection header received
0923 14:45:47.236196 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:47.236200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:47.236253 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:47.236329 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:47.237480 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.238001 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.239833 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.240276 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.240406 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.243838 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:47.243842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:47.243845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:47.243878 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:47.246437 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:47.247739 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:47.247743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:47.247744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:47.247894 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:47.248386 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:47.248392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:47.248395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:47.248673 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:47.248763 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:47.249429 (+ 666us) server_negotiation.cc:299] Negotiation successful
0923 14:45:47.249536 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:45:47.250912 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.234166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:43157 (local address 127.30.2.194:46995)
0923 14:45:47.234614 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:47.234660 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:47.235967 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:47.236480 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:47.236487 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:47.236836 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:47.237339 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.237353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.238108 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.238110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.239675 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.239690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.240451 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.240459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.240679 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.241433 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:47.241459 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:47.243619 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:47.246564 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:47.246572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:47.246588 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:47.247006 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:47.248020 (+ 1014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:47.248025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:47.248028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:47.248253 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:47.248807 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:47.248816 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:47.249152 (+ 336us) client_negotiation.cc:769] Sending connection context
0923 14:45:47.249379 (+ 227us) client_negotiation.cc:240] Negotiation successful
0923 14:45:47.249768 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260923 14:45:47.252403 2701 heartbeater.cc:347] Connected to a master server at 127.30.2.254:43157
I20260923 14:45:47.252694 2701 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:47.253240 2701 heartbeater.cc:511] Master 127.30.2.254:43157 requested a full tablet report, sending...
I20260923 14:45:47.254472 2378 ts_manager.cc:194] Registered new tserver with Master: 881e1e449a474a82bd5cddd9c34dddaf (127.30.2.194:36313)
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:45:47.255192 2378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:46995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:45:47.274928 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43157
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:47.388777 2706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:47.389024 2706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:47.389074 2706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:47.393257 2706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:47.393323 2706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:47.393383 2706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:47.398479 2706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:47.399412 2706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:47.400378 2706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:47.410291 2714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:47.410439 2706 server_base.cc:1034] running on GCE node
W20260923 14:45:47.410310 2711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:47.410704 2712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:47.411159 2706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:47.411768 2706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:47.413012 2706 hybrid_clock.cc:648] HybridClock initialized: now 1790174747412990 us; error 29 us; skew 500 ppm
Sep 23 14:45:47 dist-test-slave-wjf2 krb5kdc[2334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174747, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:47.415884 2706 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:45:47.416705 2706 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:47.416805 2706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:47.417052 2706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:47.419229 2706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "986776b6648d48959e06e787317b30fa"
format_stamp: "Formatted at 2026-09-23 14:45:47 on dist-test-slave-wjf2"
server_key: "3d03dfc0347ae9a4916e92b80a22d649"
server_key_iv: "36caa2eadc24243c99c7512e9392fb83"
server_key_version: "encryptionkey@0"
I20260923 14:45:47.419899 2706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "986776b6648d48959e06e787317b30fa"
format_stamp: "Formatted at 2026-09-23 14:45:47 on dist-test-slave-wjf2"
server_key: "3d03dfc0347ae9a4916e92b80a22d649"
server_key_iv: "36caa2eadc24243c99c7512e9392fb83"
server_key_version: "encryptionkey@0"
I20260923 14:45:47.424191 2706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 14:45:47.427024 2720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:47.428445 2706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:45:47.428615 2706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "986776b6648d48959e06e787317b30fa"
format_stamp: "Formatted at 2026-09-23 14:45:47 on dist-test-slave-wjf2"
server_key: "3d03dfc0347ae9a4916e92b80a22d649"
server_key_iv: "36caa2eadc24243c99c7512e9392fb83"
server_key_version: "encryptionkey@0"
I20260923 14:45:47.428741 2706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:47.447875 2706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:47.450826 2706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:47.451045 2706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:47.451671 2706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:47.452593 2706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:47.452670 2706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:47.452749 2706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:47.452807 2706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:47.462297 2706 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:39409
I20260923 14:45:47.462320 2833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:39409 every 8 connection(s)
I20260923 14:45:47.462916 2706 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:45:47.471300 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 2706
I20260923 14:45:47.471454 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:45:47.471771 30731 external_mini_cluster.cc:1428] Setting key 1729f5ea1e50c38ebb44b8922008fc63
Sep 23 14:45:47 dist-test-slave-wjf2 krb5kdc[2334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174747, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:47.477478 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.464685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:45693 (local address 127.30.2.254:43157)
0923 14:45:47.464833 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:47.464843 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:47.465566 (+ 723us) server_negotiation.cc:407] Connection header received
0923 14:45:47.466944 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:47.466947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:47.466996 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:47.467070 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:47.468384 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.468933 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.470226 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.470494 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.470575 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.473573 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:47.473585 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:47.473588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:47.473616 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:47.475215 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:47.475766 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:47.475771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:47.475773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:47.475900 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:47.476293 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:47.476296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:47.476298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:47.476505 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:47.476642 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:47.477171 (+ 529us) server_negotiation.cc:299] Negotiation successful
0923 14:45:47.477297 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:45:47.478343 2836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.464954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:43157 (local address 127.30.2.195:45693)
0923 14:45:47.465397 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:47.465442 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:47.466716 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:47.467283 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:47.467290 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:47.467652 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:47.468238 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.468253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.469053 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.469056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.470096 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.470106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.470648 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.470651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.470824 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.471609 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:47.471627 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:47.473385 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:47.475348 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:47.475354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:47.475371 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:47.475651 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:47.476014 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:47.476017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:47.476019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:47.476175 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:47.476642 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:47.476649 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:47.476906 (+ 257us) client_negotiation.cc:769] Sending connection context
0923 14:45:47.477123 (+ 217us) client_negotiation.cc:240] Negotiation successful
0923 14:45:47.477476 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260923 14:45:47.479496 2834 heartbeater.cc:347] Connected to a master server at 127.30.2.254:43157
I20260923 14:45:47.479750 2834 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:47.480233 2834 heartbeater.cc:511] Master 127.30.2.254:43157 requested a full tablet report, sending...
I20260923 14:45:47.481297 2378 ts_manager.cc:194] Registered new tserver with Master: 986776b6648d48959e06e787317b30fa (127.30.2.195:39409)
I20260923 14:45:47.481875 2378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:45693
I20260923 14:45:47.485724 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:45:47.495306 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.487871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.30.2.254:43157)
0923 14:45:47.488012 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:47.488017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:47.488214 (+ 197us) server_negotiation.cc:407] Connection header received
0923 14:45:47.488321 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:47.488324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:47.488364 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:47.488432 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:47.489162 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.489657 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.490482 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.490696 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.490773 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.492234 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:47.492237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:47.492239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:47.492262 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:47.493694 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:47.494067 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:47.494069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:47.494070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:47.494152 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:47.494414 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:47.494417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:47.494418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:47.494675 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:47.494802 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:47.494959 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 14:45:47.495056 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:45:47.498337 2378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:45:47.501531 2378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:47.516719 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.509948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:39409 (local address 127.0.0.1:44276)
0923 14:45:47.510945 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:47.510995 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:47.511125 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:47.511916 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:47.511923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:47.511964 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:47.512329 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.512339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.513921 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.513925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.515117 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.515126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.516111 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.516114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.516261 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.516310 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 14:45:47.516441 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 14:45:47.516563 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":163,"spinlock_wait_cycles":5632,"thread_start_us":76,"threads_started":1}
I20260923 14:45:47.517241 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.510628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.30.2.195:39409)
0923 14:45:47.511406 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:47.511416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:47.511438 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:45:47.511515 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:47.511521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:47.511722 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:47.511912 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:47.512463 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.513630 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.515247 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.515990 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.516111 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.516264 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:47.516994 (+ 730us) server_negotiation.cc:299] Negotiation successful
0923 14:45:47.517084 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":181,"threads_started":1}
I20260923 14:45:47.517274 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.510756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.30.2.194:36313)
0923 14:45:47.511278 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:47.511297 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:47.512244 (+ 947us) server_negotiation.cc:407] Connection header received
0923 14:45:47.512417 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:47.512425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:47.512612 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:47.512769 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:47.512959 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.513990 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.515535 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.516403 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.516518 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.516662 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:47.516769 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 14:45:47.516863 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":143,"threads_started":1}
I20260923 14:45:47.517813 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.510352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:36313 (local address 127.0.0.1:53958)
0923 14:45:47.512144 (+ 1792us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:47.512154 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:47.512267 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:47.512761 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:47.512763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:47.512782 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:47.512869 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.512875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.514144 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.514148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.515396 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.515407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.516517 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.516520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.516626 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.516652 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 14:45:47.516735 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 14:45:47.517664 (+ 929us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1737,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260923 14:45:47.518101 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.510062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:43761 (local address 127.0.0.1:54566)
0923 14:45:47.511090 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:47.511102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:47.511171 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:47.511916 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:47.511923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:47.511966 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:47.512329 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.512339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.515113 (+ 2774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.515117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.516311 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.516324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.517717 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.517719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.517782 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.517799 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:45:47.517892 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 14:45:47.517969 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":167,"spinlock_wait_cycles":10496,"thread_start_us":113,"threads_started":1}
I20260923 14:45:47.518173 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.510249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54566 (local address 127.30.2.193:43761)
0923 14:45:47.511406 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:47.511414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:47.511438 (+ 24us) server_negotiation.cc:407] Connection header received
0923 14:45:47.511515 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:47.511520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:47.511722 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:47.511912 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:47.513820 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.514963 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.516453 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.517412 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.517511 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.517662 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:47.517933 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 14:45:47.518019 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":104,"threads_started":1}
I20260923 14:45:47.520579 2768 tablet_service.cc:1467] Processing CreateTablet for tablet e6ad11fafd504017959320bc2168e8cc (DEFAULT_TABLE table=test-table [id=ed39ff614fa54bcd939dd76ee909e2a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:47.520815 2635 tablet_service.cc:1467] Processing CreateTablet for tablet e6ad11fafd504017959320bc2168e8cc (DEFAULT_TABLE table=test-table [id=ed39ff614fa54bcd939dd76ee909e2a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:47.521142 2502 tablet_service.cc:1467] Processing CreateTablet for tablet e6ad11fafd504017959320bc2168e8cc (DEFAULT_TABLE table=test-table [id=ed39ff614fa54bcd939dd76ee909e2a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:47.521711 2768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ad11fafd504017959320bc2168e8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:47.521905 2635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ad11fafd504017959320bc2168e8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:47.522023 2502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ad11fafd504017959320bc2168e8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:47.527894 2851 tablet_bootstrap.cc:492] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: Bootstrap starting.
I20260923 14:45:47.529064 2852 tablet_bootstrap.cc:492] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94: Bootstrap starting.
I20260923 14:45:47.529992 2853 tablet_bootstrap.cc:492] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: Bootstrap starting.
I20260923 14:45:47.530179 2851 tablet_bootstrap.cc:654] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:47.531293 2852 tablet_bootstrap.cc:654] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:47.531672 2851 log.cc:824] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:47.532647 2852 log.cc:824] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:47.532850 2853 tablet_bootstrap.cc:654] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:47.533711 2853 log.cc:824] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:47.533862 2851 tablet_bootstrap.cc:492] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: No bootstrap required, opened a new log
I20260923 14:45:47.534061 2851 ts_tablet_manager.cc:1397] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:45:47.534587 2852 tablet_bootstrap.cc:492] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94: No bootstrap required, opened a new log
I20260923 14:45:47.534765 2852 ts_tablet_manager.cc:1397] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260923 14:45:47.537199 2853 tablet_bootstrap.cc:492] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: No bootstrap required, opened a new log
I20260923 14:45:47.537400 2853 ts_tablet_manager.cc:1397] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260923 14:45:47.538015 2851 raft_consensus.cc:348] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986776b6648d48959e06e787317b30fa" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 39409 } } peers { permanent_uuid: "718e673f2812452988d3fa92b19aad94" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43761 } } peers { permanent_uuid: "881e1e449a474a82bd5cddd9c34dddaf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 36313 } }
I20260923 14:45:47.538386 2851 raft_consensus.cc:374] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:47.538482 2851 raft_consensus.cc:729] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986776b6648d48959e06e787317b30fa, State: Initialized, Role: FOLLOWER
I20260923 14:45:47.538524 2852 raft_consensus.cc:348] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986776b6648d48959e06e787317b30fa" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 39409 } } peers { permanent_uuid: "718e673f2812452988d3fa92b19aad94" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43761 } } peers { permanent_uuid: "881e1e449a474a82bd5cddd9c34dddaf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 36313 } }
I20260923 14:45:47.538756 2852 raft_consensus.cc:374] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:47.538801 2852 raft_consensus.cc:729] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718e673f2812452988d3fa92b19aad94, State: Initialized, Role: FOLLOWER
I20260923 14:45:47.538981 2851 consensus_queue.cc:260] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986776b6648d48959e06e787317b30fa" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 39409 } } peers { permanent_uuid: "718e673f2812452988d3fa92b19aad94" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43761 } } peers { permanent_uuid: "881e1e449a474a82bd5cddd9c34dddaf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 36313 } }
I20260923 14:45:47.539215 2852 consensus_queue.cc:260] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986776b6648d48959e06e787317b30fa" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 39409 } } peers { permanent_uuid: "718e673f2812452988d3fa92b19aad94" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43761 } } peers { permanent_uuid: "881e1e449a474a82bd5cddd9c34dddaf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 36313 } }
I20260923 14:45:47.539669 2834 heartbeater.cc:503] Master 127.30.2.254:43157 was elected leader, sending a full tablet report...
I20260923 14:45:47.539994 2851 ts_tablet_manager.cc:1428] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 14:45:47.540117 2852 ts_tablet_manager.cc:1428] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260923 14:45:47.540431 2568 heartbeater.cc:503] Master 127.30.2.254:43157 was elected leader, sending a full tablet report...
I20260923 14:45:47.541437 2853 raft_consensus.cc:348] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986776b6648d48959e06e787317b30fa" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 39409 } } peers { permanent_uuid: "718e673f2812452988d3fa92b19aad94" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43761 } } peers { permanent_uuid: "881e1e449a474a82bd5cddd9c34dddaf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 36313 } }
I20260923 14:45:47.541713 2853 raft_consensus.cc:374] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:47.541776 2853 raft_consensus.cc:729] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e1e449a474a82bd5cddd9c34dddaf, State: Initialized, Role: FOLLOWER
I20260923 14:45:47.542207 2853 consensus_queue.cc:260] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986776b6648d48959e06e787317b30fa" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 39409 } } peers { permanent_uuid: "718e673f2812452988d3fa92b19aad94" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43761 } } peers { permanent_uuid: "881e1e449a474a82bd5cddd9c34dddaf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 36313 } }
I20260923 14:45:47.542827 2701 heartbeater.cc:503] Master 127.30.2.254:43157 was elected leader, sending a full tablet report...
I20260923 14:45:47.543259 2853 ts_tablet_manager.cc:1428] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 14:45:47.701995 2858 raft_consensus.cc:482] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:47.702214 2858 raft_consensus.cc:504] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986776b6648d48959e06e787317b30fa" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 39409 } } peers { permanent_uuid: "718e673f2812452988d3fa92b19aad94" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43761 } } peers { permanent_uuid: "881e1e449a474a82bd5cddd9c34dddaf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 36313 } }
I20260923 14:45:47.703440 2858 leader_election.cc:288] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 986776b6648d48959e06e787317b30fa (127.30.2.195:39409), 881e1e449a474a82bd5cddd9c34dddaf (127.30.2.194:36313)
I20260923 14:45:47.708994 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.703999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:60153 (local address 127.30.2.194:36313)
0923 14:45:47.704151 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:47.704155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:47.704168 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:45:47.704315 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:47.704318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:47.704378 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:47.704450 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:47.704847 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.705631 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.707481 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.708219 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.708373 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.708435 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:47.708782 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 14:45:47.708871 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:45:47.708995 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.703822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:33245 (local address 127.30.2.195:39409)
0923 14:45:47.703967 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:47.703973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:47.704020 (+ 47us) server_negotiation.cc:407] Connection header received
0923 14:45:47.704215 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:47.704219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:47.704302 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:47.704389 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:47.704847 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.705632 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.707497 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.708267 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.708384 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.708435 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:47.708782 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 14:45:47.708866 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:45:47.708988 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.703699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:36313 (local address 127.30.2.193:60153)
0923 14:45:47.704073 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:47.704085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:47.704201 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:47.704468 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:47.704470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:47.704493 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:47.704723 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.704729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.705746 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.705750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.707314 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.707326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.708375 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.708379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.708541 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.708599 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 14:45:47.708697 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 14:45:47.708785 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":283,"spinlock_wait_cycles":2432,"thread_start_us":126,"threads_started":1}
I20260923 14:45:47.708992 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.703614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:39409 (local address 127.30.2.193:33245)
0923 14:45:47.703940 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:47.703956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:47.704066 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:47.704404 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:47.704407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:47.704440 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:47.704722 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.704729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.705746 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.705750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.707314 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:47.707326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.708375 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.708379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:47.708617 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.708651 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 14:45:47.708729 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:45:47.708787 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":171,"thread_start_us":117,"threads_started":1}
I20260923 14:45:47.709833 2788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ad11fafd504017959320bc2168e8cc" candidate_uuid: "718e673f2812452988d3fa92b19aad94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "986776b6648d48959e06e787317b30fa" is_pre_election: true
I20260923 14:45:47.709834 2655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ad11fafd504017959320bc2168e8cc" candidate_uuid: "718e673f2812452988d3fa92b19aad94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e1e449a474a82bd5cddd9c34dddaf" is_pre_election: true
I20260923 14:45:47.710158 2788 raft_consensus.cc:2454] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718e673f2812452988d3fa92b19aad94 in term 0.
I20260923 14:45:47.710175 2655 raft_consensus.cc:2454] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718e673f2812452988d3fa92b19aad94 in term 0.
I20260923 14:45:47.710693 2458 leader_election.cc:302] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 718e673f2812452988d3fa92b19aad94, 881e1e449a474a82bd5cddd9c34dddaf, 986776b6648d48959e06e787317b30fa; no voters:
I20260923 14:45:47.710978 2858 raft_consensus.cc:2781] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:47.711074 2858 raft_consensus.cc:482] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:47.711138 2858 raft_consensus.cc:3037] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:47.712256 2858 raft_consensus.cc:504] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986776b6648d48959e06e787317b30fa" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 39409 } } peers { permanent_uuid: "718e673f2812452988d3fa92b19aad94" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43761 } } peers { permanent_uuid: "881e1e449a474a82bd5cddd9c34dddaf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 36313 } }
I20260923 14:45:47.712649 2858 leader_election.cc:288] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [CANDIDATE]: Term 1 election: Requested vote from peers 986776b6648d48959e06e787317b30fa (127.30.2.195:39409), 881e1e449a474a82bd5cddd9c34dddaf (127.30.2.194:36313)
I20260923 14:45:47.712997 2788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ad11fafd504017959320bc2168e8cc" candidate_uuid: "718e673f2812452988d3fa92b19aad94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "986776b6648d48959e06e787317b30fa"
I20260923 14:45:47.713034 2655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ad11fafd504017959320bc2168e8cc" candidate_uuid: "718e673f2812452988d3fa92b19aad94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e1e449a474a82bd5cddd9c34dddaf"
I20260923 14:45:47.713126 2788 raft_consensus.cc:3037] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:47.713164 2655 raft_consensus.cc:3037] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf [term 0 FOLLOWER]: Advancing to term 1
W20260923 14:45:47.713932 2835 tablet.cc:2369] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:45:47.714320 2655 raft_consensus.cc:2454] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718e673f2812452988d3fa92b19aad94 in term 1.
I20260923 14:45:47.714346 2788 raft_consensus.cc:2454] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718e673f2812452988d3fa92b19aad94 in term 1.
I20260923 14:45:47.714633 2457 leader_election.cc:302] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718e673f2812452988d3fa92b19aad94, 986776b6648d48959e06e787317b30fa; no voters:
I20260923 14:45:47.714798 2858 raft_consensus.cc:2781] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:47.715039 2858 raft_consensus.cc:686] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [term 1 LEADER]: Becoming Leader. State: Replica: 718e673f2812452988d3fa92b19aad94, State: Running, Role: LEADER
I20260923 14:45:47.715405 2858 consensus_queue.cc:237] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986776b6648d48959e06e787317b30fa" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 39409 } } peers { permanent_uuid: "718e673f2812452988d3fa92b19aad94" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43761 } } peers { permanent_uuid: "881e1e449a474a82bd5cddd9c34dddaf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 36313 } }
I20260923 14:45:47.718827 2376 catalog_manager.cc:5495] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 718e673f2812452988d3fa92b19aad94 (127.30.2.193). New cstate: current_term: 1 leader_uuid: "718e673f2812452988d3fa92b19aad94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986776b6648d48959e06e787317b30fa" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 39409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "718e673f2812452988d3fa92b19aad94" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881e1e449a474a82bd5cddd9c34dddaf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 36313 } health_report { overall_health: UNKNOWN } } }
W20260923 14:45:47.733002 2702 tablet.cc:2369] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:45:47.749330 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.745500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.30.2.193:43761)
0923 14:45:47.745667 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:47.745673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:47.745688 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:45:47.745773 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:47.745775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:47.745829 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:47.745942 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:47.746198 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.746710 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.747649 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:47.747985 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:47.748091 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:47.748413 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:47.748840 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:47.748938 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:47.749050 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 14:45:47.749161 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:45:47.755360 2655 raft_consensus.cc:1261] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf [term 1 FOLLOWER]: Refusing update from remote peer 718e673f2812452988d3fa92b19aad94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:47.755421 2788 raft_consensus.cc:1261] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa [term 1 FOLLOWER]: Refusing update from remote peer 718e673f2812452988d3fa92b19aad94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:47.756150 2862 consensus_queue.cc:1035] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e1e449a474a82bd5cddd9c34dddaf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 36313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:47.756393 2858 consensus_queue.cc:1035] T e6ad11fafd504017959320bc2168e8cc P 718e673f2812452988d3fa92b19aad94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "986776b6648d48959e06e787317b30fa" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 39409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:47.765821 2866 mvcc.cc:205] Tried to move back new op lower bound from 7332555766798565376 to 7332555766645497856. Current Snapshot: MvccSnapshot[applied={T|T < 7332555766798565376}]
I20260923 14:45:47.773797 2378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:47.773937 2378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:47.776463 2378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a77c50035b474cbe940e276bda94853b: Sending DeleteTablet for 3 replicas of tablet e6ad11fafd504017959320bc2168e8cc
I20260923 14:45:47.777249 2768 tablet_service.cc:1514] Processing DeleteTablet for tablet e6ad11fafd504017959320bc2168e8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:47 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:44276
I20260923 14:45:47.777500 2635 tablet_service.cc:1514] Processing DeleteTablet for tablet e6ad11fafd504017959320bc2168e8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:47 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:53958
I20260923 14:45:47.777781 2502 tablet_service.cc:1514] Processing DeleteTablet for tablet e6ad11fafd504017959320bc2168e8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:47 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:54566
I20260923 14:45:47.778187 2875 tablet_replica.cc:330] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: stopping tablet replica
I20260923 14:45:47.778228 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 2440
I20260923 14:45:47.778563 2875 raft_consensus.cc:2229] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:47.779237 2875 raft_consensus.cc:2258] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:47.780269 2874 tablet_replica.cc:330] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: stopping tablet replica
I20260923 14:45:47.780561 2874 raft_consensus.cc:2229] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:47.780884 2874 raft_consensus.cc:2258] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:47.780944 2875 ts_tablet_manager.cc:1905] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:47.782174 2874 ts_tablet_manager.cc:1905] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:47.784394 2875 ts_tablet_manager.cc:1918] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:45:47.784497 2875 log.cc:1196] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/e6ad11fafd504017959320bc2168e8cc
I20260923 14:45:47.784888 2875 ts_tablet_manager.cc:1939] T e6ad11fafd504017959320bc2168e8cc P 881e1e449a474a82bd5cddd9c34dddaf: Deleting consensus metadata
I20260923 14:45:47.785156 2874 ts_tablet_manager.cc:1918] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:45:47.785254 2874 log.cc:1196] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/e6ad11fafd504017959320bc2168e8cc
I20260923 14:45:47.785663 2874 ts_tablet_manager.cc:1939] T e6ad11fafd504017959320bc2168e8cc P 986776b6648d48959e06e787317b30fa: Deleting consensus metadata
W20260923 14:45:47.785991 2362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:43761 (error 108)
I20260923 14:45:47.786064 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 2573
I20260923 14:45:47.786664 2365 catalog_manager.cc:4841] TS 881e1e449a474a82bd5cddd9c34dddaf (127.30.2.194:36313): tablet e6ad11fafd504017959320bc2168e8cc (table test-table [id=ed39ff614fa54bcd939dd76ee909e2a1]) successfully deleted
I20260923 14:45:47.788730 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.788331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:43761 (local address 127.0.0.1:54570)
0923 14:45:47.788517 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:47.788619 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:43761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 14:45:47.788940 2362 catalog_manager.cc:4584] TS 718e673f2812452988d3fa92b19aad94 (127.30.2.193:43761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6ad11fafd504017959320bc2168e8cc: Network error: Client connection negotiation failed: client connection to 127.30.2.193:43761: connect: Connection refused (error 111)
I20260923 14:45:47.789278 2364 catalog_manager.cc:4841] TS 986776b6648d48959e06e787317b30fa (127.30.2.195:39409): tablet e6ad11fafd504017959320bc2168e8cc (table test-table [id=ed39ff614fa54bcd939dd76ee909e2a1]) successfully deleted
I20260923 14:45:47.792691 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 2706
I20260923 14:45:47.799085 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 2347
2026-09-23T14:45:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2411 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T14:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:47Z Disabled control of system clock
I20260923 14:45:47.830888 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:36751
--webserver_interface=127.30.2.254
--webserver_port=0
--builtin_ntp_servers=127.30.2.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:36751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 14:45:47.942039 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:47.942271 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:47.942308 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:47.946218 2881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:47.946266 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:47.946282 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:47.946298 2881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:47.946313 2881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:47.950927 2881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:36751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:47.951967 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:47.953346 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:47.962008 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:47.962023 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:47.962023 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:47.962278 2881 server_base.cc:1034] running on GCE node
I20260923 14:45:47.962880 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:47.963984 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:47.965173 2881 hybrid_clock.cc:648] HybridClock initialized: now 1790174747965158 us; error 30 us; skew 500 ppm
I20260923 14:45:47.967330 2881 webserver.cc:460] Webserver started at http://127.30.2.254:44659/ using document root <none> and password file <none>
I20260923 14:45:47.967980 2881 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:47.968075 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:47.968309 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:47.970302 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "05903de7da4c40d09348f77b58ea23be"
format_stamp: "Formatted at 2026-09-23 14:45:47 on dist-test-slave-wjf2"
server_key: "3a406b546e02e12e0b0c405b3e0ada59"
server_key_iv: "53dcc17303e658fde516f1665bbe6f10"
server_key_version: "encryptionkey@0"
I20260923 14:45:47.970858 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "05903de7da4c40d09348f77b58ea23be"
format_stamp: "Formatted at 2026-09-23 14:45:47 on dist-test-slave-wjf2"
server_key: "3a406b546e02e12e0b0c405b3e0ada59"
server_key_iv: "53dcc17303e658fde516f1665bbe6f10"
server_key_version: "encryptionkey@0"
I20260923 14:45:47.974399 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260923 14:45:47.976898 2895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:47.978211 2881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:47.978360 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "05903de7da4c40d09348f77b58ea23be"
format_stamp: "Formatted at 2026-09-23 14:45:47 on dist-test-slave-wjf2"
server_key: "3a406b546e02e12e0b0c405b3e0ada59"
server_key_iv: "53dcc17303e658fde516f1665bbe6f10"
server_key_version: "encryptionkey@0"
I20260923 14:45:47.978472 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:47.982954 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:47.983569 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:47.983749 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:47.990381 2881 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:36751
I20260923 14:45:47.990387 2947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:36751 every 8 connection(s)
I20260923 14:45:47.991253 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:47.994376 2948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:47.996537 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 2881
I20260923 14:45:47.996701 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:47.996951 30731 external_mini_cluster.cc:1428] Setting key 106a417e4428cb0421266a711420f073
I20260923 14:45:48.000296 2948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be: Bootstrap starting.
I20260923 14:45:48.002462 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:47.998353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.30.2.254:36751)
0923 14:45:47.998992 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:47.999006 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:47.999038 (+ 32us) server_negotiation.cc:407] Connection header received
0923 14:45:47.999702 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:47.999717 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:48.000118 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:48.000434 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:48.001208 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:48.001213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:48.001242 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:48.001253 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:48.001304 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:48.001319 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:48.001459 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:48.001530 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 14:45:48.001771 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":134,"threads_started":1}
I20260923 14:45:48.002699 2948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:48.003389 2948 log.cc:824] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:48.005221 2948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be: No bootstrap required, opened a new log
I20260923 14:45:48.008955 2948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05903de7da4c40d09348f77b58ea23be" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 36751 } }
I20260923 14:45:48.009179 2948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:48.009256 2948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05903de7da4c40d09348f77b58ea23be, State: Initialized, Role: FOLLOWER
I20260923 14:45:48.009670 2948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05903de7da4c40d09348f77b58ea23be" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 36751 } }
I20260923 14:45:48.009824 2948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:48.009909 2948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:48.010026 2948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:48.010854 2948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05903de7da4c40d09348f77b58ea23be" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 36751 } }
I20260923 14:45:48.011178 2948 leader_election.cc:302] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05903de7da4c40d09348f77b58ea23be; no voters:
I20260923 14:45:48.011474 2948 leader_election.cc:288] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:48.011694 2953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:48.011956 2953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [term 1 LEADER]: Becoming Leader. State: Replica: 05903de7da4c40d09348f77b58ea23be, State: Running, Role: LEADER
I20260923 14:45:48.012267 2953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05903de7da4c40d09348f77b58ea23be" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 36751 } }
I20260923 14:45:48.012705 2948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:48.013890 2954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05903de7da4c40d09348f77b58ea23be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05903de7da4c40d09348f77b58ea23be" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 36751 } } }
I20260923 14:45:48.014046 2954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:48.013890 2955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05903de7da4c40d09348f77b58ea23be. Latest consensus state: current_term: 1 leader_uuid: "05903de7da4c40d09348f77b58ea23be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05903de7da4c40d09348f77b58ea23be" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 36751 } } }
I20260923 14:45:48.014093 2955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:48.014518 2962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:48.017588 2962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:48.022497 2962 catalog_manager.cc:1347] Generated new cluster ID: 736c075523d5465f83356760f43a4eaf
I20260923 14:45:48.022558 2962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:48.032500 2962 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:48.032792 2962 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:48.048841 2962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be: Generated new TSK 0
I20260923 14:45:48.049619 2962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:45:48.063062 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:36751
--builtin_ntp_servers=127.30.2.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 14:45:48.172211 2972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:48.172472 2972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:48.172542 2972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:48.176451 2972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:48.176517 2972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:48.176609 2972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:48.181123 2972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:48.182080 2972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:48.183367 2972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:48.192673 2977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:48.192761 2972 server_base.cc:1034] running on GCE node
W20260923 14:45:48.192673 2978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:48.192652 2980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:48.193328 2972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:48.193974 2972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:48.195170 2972 hybrid_clock.cc:648] HybridClock initialized: now 1790174748195130 us; error 52 us; skew 500 ppm
I20260923 14:45:48.197014 2972 webserver.cc:460] Webserver started at http://127.30.2.193:38819/ using document root <none> and password file <none>
I20260923 14:45:48.197580 2972 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:48.197677 2972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:48.197907 2972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:48.199971 2972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "6f593a384d044d4fb6068a45b5330149"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "4c18c9c8d933005b224b3435673c982a"
server_key_iv: "193a96f588e7d92cba8fe8f0ad1086c9"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.200516 2972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "6f593a384d044d4fb6068a45b5330149"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "4c18c9c8d933005b224b3435673c982a"
server_key_iv: "193a96f588e7d92cba8fe8f0ad1086c9"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.204008 2972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:45:48.206400 2986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:48.207566 2972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:48.207700 2972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "6f593a384d044d4fb6068a45b5330149"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "4c18c9c8d933005b224b3435673c982a"
server_key_iv: "193a96f588e7d92cba8fe8f0ad1086c9"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.207837 2972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:48.212303 2972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:48.212932 2972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:48.213117 2972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:48.213713 2972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:48.214586 2972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:48.214664 2972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:48.214731 2972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:48.214762 2972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:48.223484 2972 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:39955
I20260923 14:45:48.223521 3099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:39955 every 8 connection(s)
I20260923 14:45:48.224417 2972 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:45:48.229609 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 2972
I20260923 14:45:48.229733 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:48.230036 30731 external_mini_cluster.cc:1428] Setting key 6632e3e2f3192a7108611e1f4d16b200
I20260923 14:45:48.230716 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.226381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:60625 (local address 127.30.2.254:36751)
0923 14:45:48.226518 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:48.226522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:48.227593 (+ 1071us) server_negotiation.cc:407] Connection header received
0923 14:45:48.228933 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:48.228937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:48.228975 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:48.229052 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:48.230227 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:48.230230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:48.230233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:48.230246 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:48.230274 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:48.230284 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:48.230386 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:48.230529 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 14:45:48.230574 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:45:48.231531 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.226789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:36751 (local address 127.30.2.193:60625)
0923 14:45:48.227446 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:48.227485 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:48.228715 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:48.229237 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:48.229244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:48.229327 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:48.229940 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:45:48.229941 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:48.229979 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:45:48.230044 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:48.230371 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:48.230380 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:48.230386 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 14:45:48.230569 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 14:45:48.230800 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
I20260923 14:45:48.232291 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:36751
--builtin_ntp_servers=127.30.2.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 14:45:48.232745 3100 heartbeater.cc:347] Connected to a master server at 127.30.2.254:36751
I20260923 14:45:48.232986 3100 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:48.233428 3100 heartbeater.cc:511] Master 127.30.2.254:36751 requested a full tablet report, sending...
I20260923 14:45:48.235013 2912 ts_manager.cc:194] Registered new tserver with Master: 6f593a384d044d4fb6068a45b5330149 (127.30.2.193:39955)
W20260923 14:45:48.341845 3103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:48.342088 3103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:48.342126 3103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:48.346104 3103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:48.346160 3103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:48.346215 3103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:48.350883 3103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:48.351933 3103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:48.353288 3103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:48.362877 3108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:48.362965 3103 server_base.cc:1034] running on GCE node
W20260923 14:45:48.362921 3111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:48.362890 3109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:48.363464 3103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:48.364131 3103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:48.365329 3103 hybrid_clock.cc:648] HybridClock initialized: now 1790174748365311 us; error 25 us; skew 500 ppm
I20260923 14:45:48.367030 3103 webserver.cc:460] Webserver started at http://127.30.2.194:45715/ using document root <none> and password file <none>
I20260923 14:45:48.367604 3103 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:48.367702 3103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:48.367954 3103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:48.369971 3103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "27b0939a538f41c88d560235ac0487c2"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "4dcae08be24119b4c358e5a80d0d0415"
server_key_iv: "a6f48e9480efbaff2495b3c23bf1d971"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.370527 3103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "27b0939a538f41c88d560235ac0487c2"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "4dcae08be24119b4c358e5a80d0d0415"
server_key_iv: "a6f48e9480efbaff2495b3c23bf1d971"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.374010 3103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 14:45:48.376570 3117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:48.377916 3103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:45:48.378091 3103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "27b0939a538f41c88d560235ac0487c2"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "4dcae08be24119b4c358e5a80d0d0415"
server_key_iv: "a6f48e9480efbaff2495b3c23bf1d971"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.378217 3103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:48.383039 3103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:48.383664 3103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:48.383891 3103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:48.384456 3103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:48.385398 3103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:48.385473 3103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:48.385545 3103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:48.385598 3103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:48.395115 3103 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:46527
I20260923 14:45:48.395139 3230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:46527 every 8 connection(s)
I20260923 14:45:48.396066 3103 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:48.399693 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 3103
I20260923 14:45:48.399842 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:48.400120 30731 external_mini_cluster.cc:1428] Setting key 67e0caa1c86b339ee972cf8227272e3f
I20260923 14:45:48.402021 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:36751
--builtin_ntp_servers=127.30.2.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 14:45:48.402315 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.397943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:53285 (local address 127.30.2.254:36751)
0923 14:45:48.398099 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:48.398104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:48.399139 (+ 1035us) server_negotiation.cc:407] Connection header received
0923 14:45:48.400587 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:48.400591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:48.400638 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:48.400750 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:48.401847 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:48.401850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:48.401853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:48.401865 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:48.401888 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:48.401904 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:48.401992 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:48.402146 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 14:45:48.402189 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 14:45:48.403128 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.398355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:36751 (local address 127.30.2.194:53285)
0923 14:45:48.398987 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:48.399025 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:48.400361 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:48.400911 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:48.400918 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:48.400997 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:48.401600 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:45:48.401601 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:48.401640 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:45:48.401703 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:48.401991 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:48.402001 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:48.402009 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 14:45:48.402188 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 14:45:48.402414 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260923 14:45:48.404260 3231 heartbeater.cc:347] Connected to a master server at 127.30.2.254:36751
I20260923 14:45:48.404485 3231 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:48.404861 3231 heartbeater.cc:511] Master 127.30.2.254:36751 requested a full tablet report, sending...
I20260923 14:45:48.405879 2912 ts_manager.cc:194] Registered new tserver with Master: 27b0939a538f41c88d560235ac0487c2 (127.30.2.194:46527)
W20260923 14:45:48.512234 3234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:48.512526 3234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:48.512611 3234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:48.516929 3234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:48.517001 3234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:48.517084 3234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:48.521901 3234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:48.523099 3234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:48.524641 3234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:48.534113 3239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:48.534125 3242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:48.534245 3240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:48.534312 3234 server_base.cc:1034] running on GCE node
I20260923 14:45:48.534770 3234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:48.535351 3234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:48.536504 3234 hybrid_clock.cc:648] HybridClock initialized: now 1790174748536498 us; error 38 us; skew 500 ppm
I20260923 14:45:48.538131 3234 webserver.cc:460] Webserver started at http://127.30.2.195:41055/ using document root <none> and password file <none>
I20260923 14:45:48.538713 3234 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:48.538771 3234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:48.538928 3234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:48.540908 3234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "6286c9dee0d146979aef88ef1f8f905d"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "084e88e2448d22899e8daca8a4f27b73"
server_key_iv: "0043a870e884bfed18c39a36567f2d1e"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.541379 3234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "6286c9dee0d146979aef88ef1f8f905d"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "084e88e2448d22899e8daca8a4f27b73"
server_key_iv: "0043a870e884bfed18c39a36567f2d1e"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.544778 3234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:45:48.547017 3248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:48.548259 3234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:48.548384 3234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "6286c9dee0d146979aef88ef1f8f905d"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "084e88e2448d22899e8daca8a4f27b73"
server_key_iv: "0043a870e884bfed18c39a36567f2d1e"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.548519 3234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:48.552938 3234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:48.553498 3234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:48.553690 3234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:48.554205 3234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:48.555068 3234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:48.555138 3234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:48.555207 3234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:48.555253 3234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:48.564369 3234 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:40419
I20260923 14:45:48.564401 3361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:40419 every 8 connection(s)
I20260923 14:45:48.565259 3234 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:45:48.569087 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 3234
I20260923 14:45:48.569236 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:45:48.569562 30731 external_mini_cluster.cc:1428] Setting key 2264a2c86ea708a3b4a786828ed85159
I20260923 14:45:48.571698 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.567061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:43785 (local address 127.30.2.254:36751)
0923 14:45:48.567208 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:48.567213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:48.568301 (+ 1088us) server_negotiation.cc:407] Connection header received
0923 14:45:48.569887 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:48.569891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:48.569937 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:48.570036 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:48.571240 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:48.571244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:48.571247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:48.571259 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:48.571281 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:48.571291 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:48.571381 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:48.571506 (+ 125us) server_negotiation.cc:299] Negotiation successful
0923 14:45:48.571550 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:45:48.572552 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.567447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:36751 (local address 127.30.2.195:43785)
0923 14:45:48.568158 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:48.568197 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:48.569635 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:48.570184 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:48.570216 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:48.570323 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:48.570959 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:45:48.570961 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:48.571012 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:45:48.571101 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:48.571377 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:48.571383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:48.571388 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:45:48.571516 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 14:45:48.571748 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260923 14:45:48.573550 3362 heartbeater.cc:347] Connected to a master server at 127.30.2.254:36751
I20260923 14:45:48.573756 3362 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:48.574129 3362 heartbeater.cc:511] Master 127.30.2.254:36751 requested a full tablet report, sending...
I20260923 14:45:48.575088 2912 ts_manager.cc:194] Registered new tserver with Master: 6286c9dee0d146979aef88ef1f8f905d (127.30.2.195:40419)
I20260923 14:45:48.583565 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:45:48.586910 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.585449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.30.2.254:36751)
0923 14:45:48.585600 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:48.585603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:48.585667 (+ 64us) server_negotiation.cc:407] Connection header received
0923 14:45:48.585774 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:48.585776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:48.585805 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:48.585880 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:48.586529 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:48.586532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:48.586534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:48.586545 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:48.586561 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:48.586569 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:48.586650 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:48.586705 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 14:45:48.586736 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 14:45:48.589483 2912 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:45:48.592334 2912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:48.603269 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.600267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:46527 (local address 127.0.0.1:33944)
0923 14:45:48.600926 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:48.600977 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:48.601089 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:48.601858 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:48.601864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:48.601938 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:48.602466 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:45:48.602468 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:48.602488 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:45:48.602557 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:48.602934 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:48.602943 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:48.602949 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 14:45:48.603058 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 14:45:48.603138 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":84,"threads_started":1}
I20260923 14:45:48.603281 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.600426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.30.2.194:46527)
0923 14:45:48.601411 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:48.601425 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:48.601449 (+ 24us) server_negotiation.cc:407] Connection header received
0923 14:45:48.601511 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:48.601515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:48.601724 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:48.601870 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:48.602727 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:48.602737 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:48.602749 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:48.602760 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:48.602805 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:48.602837 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:48.602977 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:48.603072 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 14:45:48.603144 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":119,"threads_started":1}
I20260923 14:45:48.603611 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.600449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:40419 (local address 127.0.0.1:46780)
0923 14:45:48.601257 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:48.601269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:48.601337 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:48.602472 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:48.602478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:48.602498 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:48.602945 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:45:48.602947 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:48.602957 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:45:48.602983 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:48.603364 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:48.603367 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:48.603370 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:45:48.603442 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 14:45:48.603484 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":85,"threads_started":1}
I20260923 14:45:48.603945 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.601017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.30.2.195:40419)
0923 14:45:48.601783 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:48.601794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:48.601819 (+ 25us) server_negotiation.cc:407] Connection header received
0923 14:45:48.601873 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:48.601879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:48.602048 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:48.602204 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:48.603111 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:48.603119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:48.603146 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:48.603163 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:48.603204 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:48.603233 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:48.603320 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:48.603700 (+ 380us) server_negotiation.cc:299] Negotiation successful
0923 14:45:48.603770 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":113,"threads_started":1}
I20260923 14:45:48.604954 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.600762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:39955 (local address 127.0.0.1:47854)
0923 14:45:48.602673 (+ 1911us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:48.602688 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:48.602762 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:48.603815 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:48.603818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:48.603832 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:48.604286 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:45:48.604288 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:48.604298 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:45:48.604324 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:48.604725 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:48.604728 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:48.604732 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 14:45:48.604798 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 14:45:48.604832 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1836,"thread_start_us":79,"threads_started":1}
I20260923 14:45:48.605424 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.601182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.30.2.193:39955)
0923 14:45:48.601598 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:48.601612 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:48.602847 (+ 1235us) server_negotiation.cc:407] Connection header received
0923 14:45:48.602914 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:48.602922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:48.603088 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:48.603210 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:48.604465 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:48.604471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:48.604481 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:48.604494 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:48.604535 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:48.604564 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:48.604682 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:48.605217 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 14:45:48.605281 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":106,"threads_started":1}
I20260923 14:45:48.606774 3165 tablet_service.cc:1467] Processing CreateTablet for tablet 29f825d7387247ed8d48b2e6066be8e4 (DEFAULT_TABLE table=test-table [id=68ff62086eeb44beb8ed2df859ab4013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:48.606825 3296 tablet_service.cc:1467] Processing CreateTablet for tablet 29f825d7387247ed8d48b2e6066be8e4 (DEFAULT_TABLE table=test-table [id=68ff62086eeb44beb8ed2df859ab4013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:48.607752 3165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f825d7387247ed8d48b2e6066be8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:48.607856 3296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f825d7387247ed8d48b2e6066be8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:48.607954 3034 tablet_service.cc:1467] Processing CreateTablet for tablet 29f825d7387247ed8d48b2e6066be8e4 (DEFAULT_TABLE table=test-table [id=68ff62086eeb44beb8ed2df859ab4013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:48.608843 3034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f825d7387247ed8d48b2e6066be8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:48.614348 3379 tablet_bootstrap.cc:492] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: Bootstrap starting.
I20260923 14:45:48.615564 3380 tablet_bootstrap.cc:492] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d: Bootstrap starting.
I20260923 14:45:48.617105 3379 tablet_bootstrap.cc:654] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:48.617583 3380 tablet_bootstrap.cc:654] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:48.617910 3379 log.cc:824] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:48.618300 3380 log.cc:824] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:48.618564 3381 tablet_bootstrap.cc:492] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149: Bootstrap starting.
I20260923 14:45:48.619875 3379 tablet_bootstrap.cc:492] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: No bootstrap required, opened a new log
I20260923 14:45:48.620081 3379 ts_tablet_manager.cc:1397] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:45:48.620200 3380 tablet_bootstrap.cc:492] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d: No bootstrap required, opened a new log
I20260923 14:45:48.620379 3380 ts_tablet_manager.cc:1397] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:45:48.620456 3381 tablet_bootstrap.cc:654] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:48.621078 3381 log.cc:824] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:48.622957 3381 tablet_bootstrap.cc:492] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149: No bootstrap required, opened a new log
I20260923 14:45:48.623152 3381 ts_tablet_manager.cc:1397] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:45:48.624828 3380 raft_consensus.cc:348] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b0939a538f41c88d560235ac0487c2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 46527 } } peers { permanent_uuid: "6286c9dee0d146979aef88ef1f8f905d" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40419 } } peers { permanent_uuid: "6f593a384d044d4fb6068a45b5330149" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39955 } }
I20260923 14:45:48.624784 3379 raft_consensus.cc:348] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b0939a538f41c88d560235ac0487c2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 46527 } } peers { permanent_uuid: "6286c9dee0d146979aef88ef1f8f905d" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40419 } } peers { permanent_uuid: "6f593a384d044d4fb6068a45b5330149" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39955 } }
I20260923 14:45:48.625051 3379 raft_consensus.cc:374] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:48.625051 3380 raft_consensus.cc:374] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:48.625114 3379 raft_consensus.cc:729] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b0939a538f41c88d560235ac0487c2, State: Initialized, Role: FOLLOWER
I20260923 14:45:48.625118 3380 raft_consensus.cc:729] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6286c9dee0d146979aef88ef1f8f905d, State: Initialized, Role: FOLLOWER
I20260923 14:45:48.625581 3380 consensus_queue.cc:260] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b0939a538f41c88d560235ac0487c2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 46527 } } peers { permanent_uuid: "6286c9dee0d146979aef88ef1f8f905d" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40419 } } peers { permanent_uuid: "6f593a384d044d4fb6068a45b5330149" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39955 } }
I20260923 14:45:48.625581 3379 consensus_queue.cc:260] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b0939a538f41c88d560235ac0487c2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 46527 } } peers { permanent_uuid: "6286c9dee0d146979aef88ef1f8f905d" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40419 } } peers { permanent_uuid: "6f593a384d044d4fb6068a45b5330149" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39955 } }
I20260923 14:45:48.626295 3231 heartbeater.cc:503] Master 127.30.2.254:36751 was elected leader, sending a full tablet report...
I20260923 14:45:48.626572 3379 ts_tablet_manager.cc:1428] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 14:45:48.626984 3381 raft_consensus.cc:348] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b0939a538f41c88d560235ac0487c2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 46527 } } peers { permanent_uuid: "6286c9dee0d146979aef88ef1f8f905d" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40419 } } peers { permanent_uuid: "6f593a384d044d4fb6068a45b5330149" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39955 } }
I20260923 14:45:48.627197 3381 raft_consensus.cc:374] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:48.627257 3381 raft_consensus.cc:729] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f593a384d044d4fb6068a45b5330149, State: Initialized, Role: FOLLOWER
I20260923 14:45:48.627329 3362 heartbeater.cc:503] Master 127.30.2.254:36751 was elected leader, sending a full tablet report...
I20260923 14:45:48.627714 3381 consensus_queue.cc:260] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b0939a538f41c88d560235ac0487c2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 46527 } } peers { permanent_uuid: "6286c9dee0d146979aef88ef1f8f905d" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40419 } } peers { permanent_uuid: "6f593a384d044d4fb6068a45b5330149" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39955 } }
I20260923 14:45:48.628355 3100 heartbeater.cc:503] Master 127.30.2.254:36751 was elected leader, sending a full tablet report...
I20260923 14:45:48.628630 3380 ts_tablet_manager.cc:1428] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260923 14:45:48.630899 3381 ts_tablet_manager.cc:1428] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260923 14:45:48.647061 3232 tablet.cc:2369] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:45:48.714159 3386 raft_consensus.cc:482] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:48.714408 3386 raft_consensus.cc:504] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b0939a538f41c88d560235ac0487c2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 46527 } } peers { permanent_uuid: "6286c9dee0d146979aef88ef1f8f905d" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40419 } } peers { permanent_uuid: "6f593a384d044d4fb6068a45b5330149" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39955 } }
I20260923 14:45:48.715672 3386 leader_election.cc:288] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6286c9dee0d146979aef88ef1f8f905d (127.30.2.195:40419), 6f593a384d044d4fb6068a45b5330149 (127.30.2.193:39955)
I20260923 14:45:48.717738 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.716046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:54717 (local address 127.30.2.193:39955)
0923 14:45:48.716194 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:48.716199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:48.716218 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:45:48.716377 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:48.716383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:48.716446 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:48.716566 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:48.717318 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:48.717327 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:48.717329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:48.717340 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:48.717358 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:48.717366 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:48.717464 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:48.717591 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 14:45:48.717632 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:45:48.717738 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.715964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:39955 (local address 127.30.2.194:54717)
0923 14:45:48.716108 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:48.716142 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:48.716247 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:48.716606 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:48.716610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:48.716632 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:48.717135 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:45:48.717137 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:48.717147 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:45:48.717172 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:48.717458 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:48.717461 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:48.717464 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:45:48.717562 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 14:45:48.717601 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 14:45:48.717870 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.716361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:39527 (local address 127.30.2.195:40419)
0923 14:45:48.716531 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:48.716538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:48.716553 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:45:48.716598 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:48.716600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:48.716628 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:48.716714 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:48.717467 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:48.717470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:48.717473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:48.717484 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:48.717506 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:48.717520 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:48.717614 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:48.717686 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 14:45:48.717725 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 14:45:48.718088 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.715982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:40419 (local address 127.30.2.194:39527)
0923 14:45:48.716372 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:48.716383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:48.716471 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:48.716735 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:48.716738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:48.716753 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:48.717303 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:45:48.717305 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:48.717314 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:45:48.717338 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:48.717599 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:48.717602 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:48.717605 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:45:48.717690 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 14:45:48.717724 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":146,"threads_started":1}
I20260923 14:45:48.718516 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f825d7387247ed8d48b2e6066be8e4" candidate_uuid: "27b0939a538f41c88d560235ac0487c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f593a384d044d4fb6068a45b5330149" is_pre_election: true
I20260923 14:45:48.718705 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f825d7387247ed8d48b2e6066be8e4" candidate_uuid: "27b0939a538f41c88d560235ac0487c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6286c9dee0d146979aef88ef1f8f905d" is_pre_election: true
I20260923 14:45:48.718854 3054 raft_consensus.cc:2454] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b0939a538f41c88d560235ac0487c2 in term 0.
I20260923 14:45:48.718982 3316 raft_consensus.cc:2454] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b0939a538f41c88d560235ac0487c2 in term 0.
I20260923 14:45:48.719362 3120 leader_election.cc:302] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b0939a538f41c88d560235ac0487c2, 6286c9dee0d146979aef88ef1f8f905d; no voters:
I20260923 14:45:48.719664 3386 raft_consensus.cc:2781] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:48.719759 3386 raft_consensus.cc:482] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:48.719831 3386 raft_consensus.cc:3037] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:48.720841 3386 raft_consensus.cc:504] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b0939a538f41c88d560235ac0487c2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 46527 } } peers { permanent_uuid: "6286c9dee0d146979aef88ef1f8f905d" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40419 } } peers { permanent_uuid: "6f593a384d044d4fb6068a45b5330149" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39955 } }
I20260923 14:45:48.721164 3386 leader_election.cc:288] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [CANDIDATE]: Term 1 election: Requested vote from peers 6286c9dee0d146979aef88ef1f8f905d (127.30.2.195:40419), 6f593a384d044d4fb6068a45b5330149 (127.30.2.193:39955)
I20260923 14:45:48.721493 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f825d7387247ed8d48b2e6066be8e4" candidate_uuid: "27b0939a538f41c88d560235ac0487c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f593a384d044d4fb6068a45b5330149"
I20260923 14:45:48.721494 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f825d7387247ed8d48b2e6066be8e4" candidate_uuid: "27b0939a538f41c88d560235ac0487c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6286c9dee0d146979aef88ef1f8f905d"
I20260923 14:45:48.721613 3316 raft_consensus.cc:3037] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:48.721613 3054 raft_consensus.cc:3037] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:48.722903 3054 raft_consensus.cc:2454] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b0939a538f41c88d560235ac0487c2 in term 1.
I20260923 14:45:48.722903 3316 raft_consensus.cc:2454] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b0939a538f41c88d560235ac0487c2 in term 1.
I20260923 14:45:48.723206 3120 leader_election.cc:302] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 27b0939a538f41c88d560235ac0487c2, 6286c9dee0d146979aef88ef1f8f905d, 6f593a384d044d4fb6068a45b5330149; no voters:
I20260923 14:45:48.723397 3386 raft_consensus.cc:2781] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:48.723572 3386 raft_consensus.cc:686] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 1 LEADER]: Becoming Leader. State: Replica: 27b0939a538f41c88d560235ac0487c2, State: Running, Role: LEADER
I20260923 14:45:48.723943 3386 consensus_queue.cc:237] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b0939a538f41c88d560235ac0487c2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 46527 } } peers { permanent_uuid: "6286c9dee0d146979aef88ef1f8f905d" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40419 } } peers { permanent_uuid: "6f593a384d044d4fb6068a45b5330149" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39955 } }
W20260923 14:45:48.725561 3101 tablet.cc:2369] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:45:48.727137 2912 catalog_manager.cc:5495] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27b0939a538f41c88d560235ac0487c2 (127.30.2.194). New cstate: current_term: 1 leader_uuid: "27b0939a538f41c88d560235ac0487c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b0939a538f41c88d560235ac0487c2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 46527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6286c9dee0d146979aef88ef1f8f905d" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f593a384d044d4fb6068a45b5330149" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39955 } health_report { overall_health: UNKNOWN } } }
I20260923 14:45:48.756856 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.755299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.30.2.194:46527)
0923 14:45:48.755449 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:48.755453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:48.755468 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:45:48.755547 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:48.755550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:48.755625 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:48.755698 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:48.756462 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:48.756466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:48.756467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:48.756477 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:48.756496 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:48.756504 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:48.756594 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:48.756641 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 14:45:48.756676 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 14:45:48.763100 3054 raft_consensus.cc:1261] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149 [term 1 FOLLOWER]: Refusing update from remote peer 27b0939a538f41c88d560235ac0487c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:48.763101 3316 raft_consensus.cc:1261] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d [term 1 FOLLOWER]: Refusing update from remote peer 27b0939a538f41c88d560235ac0487c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:48.763906 3386 consensus_queue.cc:1035] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f593a384d044d4fb6068a45b5330149" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:48.764081 3389 consensus_queue.cc:1035] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6286c9dee0d146979aef88ef1f8f905d" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:48.771230 3392 mvcc.cc:205] Tried to move back new op lower bound from 7332555770925719552 to 7332555770776170496. Current Snapshot: MvccSnapshot[applied={T|T < 7332555770925719552}]
I20260923 14:45:48.772050 3394 mvcc.cc:205] Tried to move back new op lower bound from 7332555770925719552 to 7332555770776170496. Current Snapshot: MvccSnapshot[applied={T|T < 7332555770925719552}]
I20260923 14:45:48.780503 2912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:48.780689 2912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:48.783339 2912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05903de7da4c40d09348f77b58ea23be: Sending DeleteTablet for 3 replicas of tablet 29f825d7387247ed8d48b2e6066be8e4
I20260923 14:45:48.784157 3165 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f825d7387247ed8d48b2e6066be8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:48 UTC) from {username='slave'} at 127.0.0.1:33944
I20260923 14:45:48.784437 3034 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f825d7387247ed8d48b2e6066be8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:48 UTC) from {username='slave'} at 127.0.0.1:47854
I20260923 14:45:48.784694 3401 tablet_replica.cc:330] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: stopping tablet replica
I20260923 14:45:48.784721 3296 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f825d7387247ed8d48b2e6066be8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:48 UTC) from {username='slave'} at 127.0.0.1:46780
I20260923 14:45:48.785028 3401 raft_consensus.cc:2229] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:45:48.785279 3403 tablet_replica.cc:330] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d: stopping tablet replica
I20260923 14:45:48.785410 3401 raft_consensus.cc:2258] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:48.785612 3402 tablet_replica.cc:330] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149: stopping tablet replica
I20260923 14:45:48.785908 3402 raft_consensus.cc:2229] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:48.786023 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 2972
I20260923 14:45:48.786267 3402 raft_consensus.cc:2258] T 29f825d7387247ed8d48b2e6066be8e4 P 6f593a384d044d4fb6068a45b5330149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:48.787041 3401 ts_tablet_manager.cc:1905] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:48.789675 3401 ts_tablet_manager.cc:1918] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:45:48.789750 3401 log.cc:1196] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/29f825d7387247ed8d48b2e6066be8e4
I20260923 14:45:48.790009 3401 ts_tablet_manager.cc:1939] T 29f825d7387247ed8d48b2e6066be8e4 P 27b0939a538f41c88d560235ac0487c2: Deleting consensus metadata
I20260923 14:45:48.791152 2899 catalog_manager.cc:4841] TS 27b0939a538f41c88d560235ac0487c2 (127.30.2.194:46527): tablet 29f825d7387247ed8d48b2e6066be8e4 (table test-table [id=68ff62086eeb44beb8ed2df859ab4013]) successfully deleted
I20260923 14:45:48.792055 3403 raft_consensus.cc:2229] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:48.792410 3403 raft_consensus.cc:2258] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 14:45:48.795568 2897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:39955 (error 108)
I20260923 14:45:48.795712 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 3103
I20260923 14:45:48.797178 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:48.796799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:39955 (local address 127.0.0.1:47862)
0923 14:45:48.797007 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:48.797084 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:39955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260923 14:45:48.797183 3403 ts_tablet_manager.cc:1905] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 14:45:48.797487 2897 catalog_manager.cc:4584] TS 6f593a384d044d4fb6068a45b5330149 (127.30.2.193:39955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29f825d7387247ed8d48b2e6066be8e4: Network error: Client connection negotiation failed: client connection to 127.30.2.193:39955: connect: Connection refused (error 111)
I20260923 14:45:48.799526 3403 ts_tablet_manager.cc:1918] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:45:48.799650 3403 log.cc:1196] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/29f825d7387247ed8d48b2e6066be8e4
I20260923 14:45:48.800021 3403 ts_tablet_manager.cc:1939] T 29f825d7387247ed8d48b2e6066be8e4 P 6286c9dee0d146979aef88ef1f8f905d: Deleting consensus metadata
I20260923 14:45:48.801417 2898 catalog_manager.cc:4841] TS 6286c9dee0d146979aef88ef1f8f905d (127.30.2.195:40419): tablet 29f825d7387247ed8d48b2e6066be8e4 (table test-table [id=68ff62086eeb44beb8ed2df859ab4013]) successfully deleted
I20260923 14:45:48.803333 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 3234
I20260923 14:45:48.809851 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 2881
2026-09-23T14:45:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1009 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 14:45:48.832206 30731 mini_oidc.cc:152] Starting JWKS server
I20260923 14:45:48.833773 30731 webserver.cc:460] Webserver started at https://127.0.0.1:39697/ using document root <none> and password file <none>
I20260923 14:45:48.833905 30731 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 14:45:48.834120 30731 webserver.cc:460] Webserver started at http://127.0.0.1:41315/ using document root <none> and password file <none>
2026-09-23T14:45:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:48Z Disabled control of system clock
I20260923 14:45:48.843250 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:37715
--webserver_interface=127.30.2.254
--webserver_port=0
--builtin_ntp_servers=127.30.2.212:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:37715
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39697/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/testchainca.cert with env {}
W20260923 14:45:48.955359 3410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:48.955595 3410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:48.955632 3410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:48.959600 3410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:48.959651 3410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:48.959666 3410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:48.959683 3410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:48.959697 3410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 14:45:48.959712 3410 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 14:45:48.959725 3410 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39697/jwks.json
I20260923 14:45:48.964787 3410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:37715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:37715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39697/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:48.965696 3410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:48.966593 3410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:48.975008 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:48.975033 3418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:48.975157 3410 server_base.cc:1034] running on GCE node
W20260923 14:45:48.975008 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:48.976002 3410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:48.977048 3410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:48.978252 3410 hybrid_clock.cc:648] HybridClock initialized: now 1790174748978228 us; error 30 us; skew 500 ppm
I20260923 14:45:48.980214 3410 webserver.cc:460] Webserver started at http://127.30.2.254:35105/ using document root <none> and password file <none>
I20260923 14:45:48.980798 3410 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:48.980894 3410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:48.981149 3410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:48.983178 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "5651cd0e70d842c6917cf45de6fc41f0"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "b3dd07a227202bd0ec1bf4f81f6b348b"
server_key_iv: "c5a6a96c5e7bca9e53ef62c963bfed38"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.983927 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "5651cd0e70d842c6917cf45de6fc41f0"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "b3dd07a227202bd0ec1bf4f81f6b348b"
server_key_iv: "c5a6a96c5e7bca9e53ef62c963bfed38"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.987582 3410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:45:48.990022 3424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:48.991165 3410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:48.991312 3410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "5651cd0e70d842c6917cf45de6fc41f0"
format_stamp: "Formatted at 2026-09-23 14:45:48 on dist-test-slave-wjf2"
server_key: "b3dd07a227202bd0ec1bf4f81f6b348b"
server_key_iv: "c5a6a96c5e7bca9e53ef62c963bfed38"
server_key_version: "encryptionkey@0"
I20260923 14:45:48.991430 3410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:49.007181 3410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:49.007887 3410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:49.008111 3410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:49.015470 3410 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:37715
I20260923 14:45:49.015472 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:37715 every 8 connection(s)
I20260923 14:45:49.016463 3410 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:49.018595 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 3410
I20260923 14:45:49.018754 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:49.019073 30731 external_mini_cluster.cc:1428] Setting key 99f72d880d0a01fac631ded235411ea1
I20260923 14:45:49.019932 3477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:49.026743 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0: Bootstrap starting.
I20260923 14:45:49.029552 3477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:49.029608 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:49.020440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57438 (local address 127.30.2.254:37715)
0923 14:45:49.021107 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:49.021127 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:49.021170 (+ 43us) server_negotiation.cc:407] Connection header received
0923 14:45:49.022037 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:49.022059 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:49.022538 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:49.022914 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:49.023353 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.024791 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.025855 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.026510 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.026684 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:49.027547 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:49.027554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:49.027568 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:49.028090 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:49.028161 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:49.028193 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:49.028292 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:49.028452 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 14:45:49.028716 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20260923 14:45:49.030341 3477 log.cc:824] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:49.032296 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:45:49.032398 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0: No bootstrap required, opened a new log
I20260923 14:45:49.035949 3477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5651cd0e70d842c6917cf45de6fc41f0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37715 } }
I20260923 14:45:49.036185 3477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:49.036266 3477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5651cd0e70d842c6917cf45de6fc41f0, State: Initialized, Role: FOLLOWER
I20260923 14:45:49.036665 3477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5651cd0e70d842c6917cf45de6fc41f0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37715 } }
I20260923 14:45:49.036804 3477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:49.036885 3477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:49.036998 3477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:49.037858 3477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5651cd0e70d842c6917cf45de6fc41f0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37715 } }
I20260923 14:45:49.038189 3477 leader_election.cc:302] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5651cd0e70d842c6917cf45de6fc41f0; no voters:
I20260923 14:45:49.038493 3477 leader_election.cc:288] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:49.038622 3482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:49.038890 3482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [term 1 LEADER]: Becoming Leader. State: Replica: 5651cd0e70d842c6917cf45de6fc41f0, State: Running, Role: LEADER
I20260923 14:45:49.039206 3482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5651cd0e70d842c6917cf45de6fc41f0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37715 } }
I20260923 14:45:49.039728 3477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:49.040978 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5651cd0e70d842c6917cf45de6fc41f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5651cd0e70d842c6917cf45de6fc41f0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37715 } } }
I20260923 14:45:49.041007 3483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5651cd0e70d842c6917cf45de6fc41f0. Latest consensus state: current_term: 1 leader_uuid: "5651cd0e70d842c6917cf45de6fc41f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5651cd0e70d842c6917cf45de6fc41f0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37715 } } }
I20260923 14:45:49.041101 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:49.041126 3483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:49.041494 3491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:49.044576 3491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:49.049860 3491 catalog_manager.cc:1347] Generated new cluster ID: 77595acc7e6c404cb39eb5274ff13641
I20260923 14:45:49.049949 3491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:49.063994 3491 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:49.065057 3491 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:49.073413 3491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5651cd0e70d842c6917cf45de6fc41f0: Generated new TSK 0
I20260923 14:45:49.074158 3491 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 14:45:49.109661 3507 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 14:45:49.109771 3507 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 14:45:49.110321 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:49.093377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.30.2.254:37715)
0923 14:45:49.093531 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:49.093537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:49.093793 (+ 256us) server_negotiation.cc:407] Connection header received
0923 14:45:49.094022 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:49.094025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:49.094069 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:49.094154 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0923 14:45:49.094456 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.095046 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.096132 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.096371 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.096483 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:49.096825 (+ 342us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 14:45:49.109935 (+ 13110us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 14:45:49.110188 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 14:45:49.110405 3480 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 14:45:49.111578 3506 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:45:49.093321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:37715 (local address 127.0.0.1:57450)
0923 14:45:49.093700 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:49.093714 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:49.093841 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:49.094196 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:49.094199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:49.094213 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0923 14:45:49.094323 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:49.094330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.095186 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.095191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:49.095984 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:49.095993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.096499 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.096503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:49.096595 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:49.096636 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 14:45:49.110129 (+ 13493us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 14:45:49.110332 (+ 203us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.2.254:37715: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":146,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.30.2.254:37715: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 14:45:49.112231 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 3410
2026-09-23T14:45:49Z chronyd exiting
I20260923 14:45:49.150593 30731 test_util.cc:175] -----------------------------------------------
I20260923 14:45:49.150689 30731 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174723338690-30731-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (320 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 14:45:49.151924 30731 mini_oidc.cc:152] Starting JWKS server
I20260923 14:45:49.152894 30731 webserver.cc:460] Webserver started at https://127.0.0.1:40445/ using document root <none> and password file <none>
I20260923 14:45:49.153002 30731 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 14:45:49.153259 30731 webserver.cc:460] Webserver started at http://127.0.0.1:43021/ using document root <none> and password file <none>
2026-09-23T14:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:49Z Disabled control of system clock
I20260923 14:45:49.162717 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:32785
--webserver_interface=127.30.2.254
--webserver_port=0
--builtin_ntp_servers=127.30.2.212:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:32785
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40445/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/testchainca.cert with env {}
W20260923 14:45:49.288662 3513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:49.288918 3513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:49.288959 3513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:49.292867 3513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:49.292917 3513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:49.292949 3513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:49.292968 3513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:49.292985 3513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 14:45:49.293004 3513 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 14:45:49.293025 3513 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40445/jwks.json
I20260923 14:45:49.298184 3513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:32785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:32785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40445/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:49.299209 3513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:49.300297 3513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:49.308964 3518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:49.309087 3513 server_base.cc:1034] running on GCE node
W20260923 14:45:49.309053 3519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:49.308981 3521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:49.309937 3513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:49.310997 3513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:49.312155 3513 hybrid_clock.cc:648] HybridClock initialized: now 1790174749312135 us; error 31 us; skew 500 ppm
I20260923 14:45:49.313884 3513 webserver.cc:460] Webserver started at http://127.30.2.254:39215/ using document root <none> and password file <none>
I20260923 14:45:49.314422 3513 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:49.314477 3513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:49.314642 3513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:49.316658 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "cb183c3a48884cb690b05853f1520f76"
format_stamp: "Formatted at 2026-09-23 14:45:49 on dist-test-slave-wjf2"
server_key: "ba3c419e5489a4bdf881133d5f41c7e5"
server_key_iv: "833a9f96722d1fad4e2f13ed98275e5f"
server_key_version: "encryptionkey@0"
I20260923 14:45:49.317150 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "cb183c3a48884cb690b05853f1520f76"
format_stamp: "Formatted at 2026-09-23 14:45:49 on dist-test-slave-wjf2"
server_key: "ba3c419e5489a4bdf881133d5f41c7e5"
server_key_iv: "833a9f96722d1fad4e2f13ed98275e5f"
server_key_version: "encryptionkey@0"
I20260923 14:45:49.320851 3513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260923 14:45:49.323320 3527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:49.324648 3513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:49.324781 3513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "cb183c3a48884cb690b05853f1520f76"
format_stamp: "Formatted at 2026-09-23 14:45:49 on dist-test-slave-wjf2"
server_key: "ba3c419e5489a4bdf881133d5f41c7e5"
server_key_iv: "833a9f96722d1fad4e2f13ed98275e5f"
server_key_version: "encryptionkey@0"
I20260923 14:45:49.324887 3513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:49.342257 3513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:49.342909 3513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:49.343101 3513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:49.350281 3513 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:32785
I20260923 14:45:49.350287 3579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:32785 every 8 connection(s)
I20260923 14:45:49.351203 3513 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:49.354444 3580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:49.359434 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 3513
I20260923 14:45:49.359606 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:49.359824 3580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76: Bootstrap starting.
I20260923 14:45:49.360031 30731 external_mini_cluster.cc:1428] Setting key 90166bb47ea38e97d2ab3917756bedcf
I20260923 14:45:49.362419 3580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:49.363169 3580 log.cc:824] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:49.365132 3580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76: No bootstrap required, opened a new log
I20260923 14:45:49.369414 3580 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb183c3a48884cb690b05853f1520f76" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32785 } }
I20260923 14:45:49.369663 3580 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:49.369740 3580 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb183c3a48884cb690b05853f1520f76, State: Initialized, Role: FOLLOWER
I20260923 14:45:49.370062 3583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:49.361377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.30.2.254:32785)
0923 14:45:49.362044 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:49.362059 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:49.362097 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:45:49.362845 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:49.362866 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:49.363326 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:49.363721 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:49.364217 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.365300 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.366524 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.367044 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.367167 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:49.368063 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:49.368069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:49.368083 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:49.368484 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:49.368541 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:49.368562 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:49.368645 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:49.368846 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 14:45:49.369152 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20260923 14:45:49.370249 3580 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb183c3a48884cb690b05853f1520f76" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32785 } }
I20260923 14:45:49.370407 3580 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:49.370472 3580 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:49.370564 3580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:49.371505 3580 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb183c3a48884cb690b05853f1520f76" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32785 } }
I20260923 14:45:49.371917 3580 leader_election.cc:302] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb183c3a48884cb690b05853f1520f76; no voters:
I20260923 14:45:49.372239 3580 leader_election.cc:288] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:49.372371 3585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:49.372648 3585 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [term 1 LEADER]: Becoming Leader. State: Replica: cb183c3a48884cb690b05853f1520f76, State: Running, Role: LEADER
I20260923 14:45:49.372958 3585 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb183c3a48884cb690b05853f1520f76" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32785 } }
I20260923 14:45:49.373204 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:45:49.373483 3580 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:49.374621 3586 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb183c3a48884cb690b05853f1520f76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb183c3a48884cb690b05853f1520f76" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32785 } } }
I20260923 14:45:49.374754 3586 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:49.375124 3594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:49.375432 3587 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb183c3a48884cb690b05853f1520f76. Latest consensus state: current_term: 1 leader_uuid: "cb183c3a48884cb690b05853f1520f76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb183c3a48884cb690b05853f1520f76" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32785 } } }
I20260923 14:45:49.375533 3587 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:49.378119 3594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:49.383594 3594 catalog_manager.cc:1347] Generated new cluster ID: bd97a8bc28914c7fa63a9a70185c703b
I20260923 14:45:49.383679 3594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:49.399415 3594 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:49.400568 3594 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:49.419224 3594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb183c3a48884cb690b05853f1520f76: Generated new TSK 0
I20260923 14:45:49.420018 3594 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 14:45:49.449990 3610 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 14:45:49.450073 3610 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 14:45:49.450788 3583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:49.434908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.30.2.254:32785)
0923 14:45:49.435026 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:49.435030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:49.435221 (+ 191us) server_negotiation.cc:407] Connection header received
0923 14:45:49.435422 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:49.435425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:49.435471 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:49.435584 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0923 14:45:49.435856 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.436393 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.437266 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.437529 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.437644 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:49.437974 (+ 330us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 14:45:49.450374 (+ 12400us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40445/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 14:45:49.450636 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45652: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40445/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":43}
W20260923 14:45:49.450857 3583 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45652: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40445/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 14:45:49.450886 3609 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:45:49.434835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32785 (local address 127.0.0.1:45652)
0923 14:45:49.435130 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:49.435144 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:49.435273 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:49.435583 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:49.435586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:49.435594 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0923 14:45:49.435714 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:49.435727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.436509 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.436513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:49.437147 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:49.437155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.437663 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.437667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:49.437775 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:49.437797 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 14:45:49.450556 (+ 12759us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40445/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 14:45:49.450716 (+ 160us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.2.254:32785: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40445/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":135,"threads_started":1}
I20260923 14:45:49.451720 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 3513
2026-09-23T14:45:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (339 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 14:45:49.491727 30731 mini_oidc.cc:152] Starting JWKS server
I20260923 14:45:49.492635 30731 webserver.cc:460] Webserver started at https://127.0.0.1:33413/ using document root <none> and password file <none>
I20260923 14:45:49.492741 30731 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 14:45:49.493002 30731 webserver.cc:460] Webserver started at http://127.0.0.1:42229/ using document root <none> and password file <none>
2026-09-23T14:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:49Z Disabled control of system clock
I20260923 14:45:49.501819 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:40117
--webserver_interface=127.30.2.254
--webserver_port=0
--builtin_ntp_servers=127.30.2.212:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:40117
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33413/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 14:45:49.608881 3616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:49.609107 3616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:49.609143 3616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:49.613087 3616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:49.613135 3616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:49.613150 3616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:49.613166 3616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:49.613181 3616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 14:45:49.613231 3616 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 14:45:49.613258 3616 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33413/jwks.json
I20260923 14:45:49.617892 3616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:40117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33413/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:49.618873 3616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:49.620270 3616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:49.628510 3622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:49.628533 3624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:49.628697 3616 server_base.cc:1034] running on GCE node
W20260923 14:45:49.628839 3621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:49.629348 3616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:49.630432 3616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:49.631662 3616 hybrid_clock.cc:648] HybridClock initialized: now 1790174749631644 us; error 25 us; skew 500 ppm
I20260923 14:45:49.633555 3616 webserver.cc:460] Webserver started at http://127.30.2.254:43713/ using document root <none> and password file <none>
I20260923 14:45:49.634137 3616 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:49.634230 3616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:49.634464 3616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:49.636611 3616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "06ccf61d35ec43d386e70339ff895d77"
format_stamp: "Formatted at 2026-09-23 14:45:49 on dist-test-slave-wjf2"
server_key: "4a833936bca23d2a69f908054c93d601"
server_key_iv: "12b115d28304a8258377e3c08b47ad5f"
server_key_version: "encryptionkey@0"
I20260923 14:45:49.637184 3616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "06ccf61d35ec43d386e70339ff895d77"
format_stamp: "Formatted at 2026-09-23 14:45:49 on dist-test-slave-wjf2"
server_key: "4a833936bca23d2a69f908054c93d601"
server_key_iv: "12b115d28304a8258377e3c08b47ad5f"
server_key_version: "encryptionkey@0"
I20260923 14:45:49.640813 3616 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:45:49.643258 3630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:49.644577 3616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:45:49.644716 3616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "06ccf61d35ec43d386e70339ff895d77"
format_stamp: "Formatted at 2026-09-23 14:45:49 on dist-test-slave-wjf2"
server_key: "4a833936bca23d2a69f908054c93d601"
server_key_iv: "12b115d28304a8258377e3c08b47ad5f"
server_key_version: "encryptionkey@0"
I20260923 14:45:49.644829 3616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:49.669041 3616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:49.669761 3616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:49.669967 3616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:49.677940 3616 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:40117
I20260923 14:45:49.677964 3682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:40117 every 8 connection(s)
I20260923 14:45:49.678874 3616 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:49.682013 3683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:49.687151 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 3616
I20260923 14:45:49.687281 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:49.687561 30731 external_mini_cluster.cc:1428] Setting key 60a9131c9688170043d3222f66b9fc2b
I20260923 14:45:49.687629 3683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77: Bootstrap starting.
I20260923 14:45:49.690204 3683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:49.690977 3683 log.cc:824] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:49.692941 3683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77: No bootstrap required, opened a new log
I20260923 14:45:49.696590 3683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ccf61d35ec43d386e70339ff895d77" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40117 } }
I20260923 14:45:49.696791 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:49.688914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.30.2.254:40117)
0923 14:45:49.689542 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:49.689556 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:49.689593 (+ 37us) server_negotiation.cc:407] Connection header received
0923 14:45:49.690197 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:49.690214 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:49.690654 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:49.690982 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:49.691460 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.692553 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.693629 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.694148 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.694313 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:49.695177 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:49.695183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:49.695195 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:49.695559 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:49.695614 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:49.695634 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:49.695754 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:49.695890 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 14:45:49.696135 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260923 14:45:49.696810 3683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:49.696954 3683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ccf61d35ec43d386e70339ff895d77, State: Initialized, Role: FOLLOWER
I20260923 14:45:49.697430 3683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ccf61d35ec43d386e70339ff895d77" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40117 } }
I20260923 14:45:49.697566 3683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:49.697615 3683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:49.697695 3683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:49.698505 3683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ccf61d35ec43d386e70339ff895d77" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40117 } }
I20260923 14:45:49.698866 3683 leader_election.cc:302] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06ccf61d35ec43d386e70339ff895d77; no voters:
I20260923 14:45:49.699265 3683 leader_election.cc:288] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:49.699267 3688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:49.699479 3688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [term 1 LEADER]: Becoming Leader. State: Replica: 06ccf61d35ec43d386e70339ff895d77, State: Running, Role: LEADER
I20260923 14:45:49.699895 3688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ccf61d35ec43d386e70339ff895d77" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40117 } }
I20260923 14:45:49.700230 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:45:49.700634 3683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:49.701103 3690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06ccf61d35ec43d386e70339ff895d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ccf61d35ec43d386e70339ff895d77" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40117 } } }
I20260923 14:45:49.701236 3690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:49.701891 3689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06ccf61d35ec43d386e70339ff895d77. Latest consensus state: current_term: 1 leader_uuid: "06ccf61d35ec43d386e70339ff895d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ccf61d35ec43d386e70339ff895d77" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40117 } } }
I20260923 14:45:49.701979 3692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:49.702015 3689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:49.705156 3692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:49.710451 3692 catalog_manager.cc:1347] Generated new cluster ID: 24adce33fb0f4797acbe768a0117ec04
I20260923 14:45:49.710520 3692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:49.731381 3692 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:49.732435 3692 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:49.739319 3692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06ccf61d35ec43d386e70339ff895d77: Generated new TSK 0
I20260923 14:45:49.740211 3692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:45:49.758956 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:49.753800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.30.2.254:40117)
0923 14:45:49.753949 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:49.753954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:49.754178 (+ 224us) server_negotiation.cc:407] Connection header received
0923 14:45:49.754291 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:49.754293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:49.754347 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:49.754436 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0923 14:45:49.754697 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.755236 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.756363 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.756713 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.756819 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:49.757146 (+ 327us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 14:45:49.758597 (+ 1451us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 14:45:49.758818 (+ 221us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47594: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 14:45:49.759028 3686 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47594: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 14:45:49.760696 3712 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:45:49.753720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40117 (local address 127.0.0.1:47594)
0923 14:45:49.754041 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:49.754055 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:49.754181 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:49.754461 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:49.754465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:49.754474 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0923 14:45:49.754591 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:49.754598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.755347 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.755351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:49.756222 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:49.756233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.756864 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.756867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:49.756935 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:49.756958 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 14:45:49.758752 (+ 1794us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 14:45:49.760528 (+ 1776us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.2.254:40117: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":1593,"thread_start_us":120,"threads_started":1}
I20260923 14:45:49.760977 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 3616
E20260923 14:45:49.769465 3713 webserver.cc:533] Webserver: sslize failed:
E20260923 14:45:49.769528 3713 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-23T14:45:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (308 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T14:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:49Z Disabled control of system clock
I20260923 14:45:49.808897 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:35505
--webserver_interface=127.30.2.254
--webserver_port=0
--builtin_ntp_servers=127.30.2.212:43259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:35505
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/testchainca.cert with env {}
W20260923 14:45:49.919445 3717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:49.919672 3717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:49.919706 3717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:49.923755 3717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:49.923816 3717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:49.923832 3717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:49.923848 3717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:49.923863 3717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 14:45:49.923878 3717 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 14:45:49.923892 3717 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 14:45:49.929212 3717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:43259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:35505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:35505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:49.930150 3717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:49.931109 3717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:49.939507 3725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:49.939531 3723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:49.939564 3722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:49.939661 3717 server_base.cc:1034] running on GCE node
I20260923 14:45:49.940292 3717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:49.941347 3717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:49.942569 3717 hybrid_clock.cc:648] HybridClock initialized: now 1790174749942540 us; error 38 us; skew 500 ppm
I20260923 14:45:49.944520 3717 webserver.cc:460] Webserver started at http://127.30.2.254:45981/ using document root <none> and password file <none>
I20260923 14:45:49.945133 3717 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:49.945226 3717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:49.945456 3717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:49.947744 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "6ca0deef7b3b49bd95c137498893321d"
format_stamp: "Formatted at 2026-09-23 14:45:49 on dist-test-slave-wjf2"
server_key: "50476c0346fd616127f24664b725c2ae"
server_key_iv: "4ad3008ae7786392e4bba9f3fd1ef64f"
server_key_version: "encryptionkey@0"
I20260923 14:45:49.948379 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "6ca0deef7b3b49bd95c137498893321d"
format_stamp: "Formatted at 2026-09-23 14:45:49 on dist-test-slave-wjf2"
server_key: "50476c0346fd616127f24664b725c2ae"
server_key_iv: "4ad3008ae7786392e4bba9f3fd1ef64f"
server_key_version: "encryptionkey@0"
I20260923 14:45:49.951864 3717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:45:49.954250 3731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:49.955474 3717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:49.955619 3717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "6ca0deef7b3b49bd95c137498893321d"
format_stamp: "Formatted at 2026-09-23 14:45:49 on dist-test-slave-wjf2"
server_key: "50476c0346fd616127f24664b725c2ae"
server_key_iv: "4ad3008ae7786392e4bba9f3fd1ef64f"
server_key_version: "encryptionkey@0"
I20260923 14:45:49.955739 3717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:49.968464 3717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:49.969048 3717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:49.969233 3717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:49.976251 3717 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:35505
I20260923 14:45:49.976274 3783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:35505 every 8 connection(s)
I20260923 14:45:49.977175 3717 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:49.980039 3784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:49.984292 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 3717
I20260923 14:45:49.984416 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:49.984715 30731 external_mini_cluster.cc:1428] Setting key 7a6d46296cd74b4b0dd86c4e9d0fe884
I20260923 14:45:49.985828 3784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d: Bootstrap starting.
I20260923 14:45:49.988023 3784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:49.988883 3784 log.cc:824] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:49.990834 3784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d: No bootstrap required, opened a new log
I20260923 14:45:49.994199 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:49.986113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.30.2.254:35505)
0923 14:45:49.986742 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:49.986758 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:49.986811 (+ 53us) server_negotiation.cc:407] Connection header received
0923 14:45:49.987465 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:49.987483 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:49.987965 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:49.988302 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:49.988662 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.989752 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.990924 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:49.991389 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:49.991503 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:49.992361 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:49.992366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:49.992379 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:45:49.992772 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:49.992829 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:45:49.992851 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:49.992939 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:49.993113 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 14:45:49.993422 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":132,"threads_started":1}
I20260923 14:45:49.994730 3784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca0deef7b3b49bd95c137498893321d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35505 } }
I20260923 14:45:49.994930 3784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:49.994985 3784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca0deef7b3b49bd95c137498893321d, State: Initialized, Role: FOLLOWER
I20260923 14:45:49.995452 3784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca0deef7b3b49bd95c137498893321d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35505 } }
I20260923 14:45:49.995602 3784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:49.995677 3784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:49.995766 3784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:49.996758 3784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca0deef7b3b49bd95c137498893321d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35505 } }
I20260923 14:45:49.997120 3784 leader_election.cc:302] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ca0deef7b3b49bd95c137498893321d; no voters:
I20260923 14:45:49.997327 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:45:49.997500 3784 leader_election.cc:288] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:49.997593 3789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:49.997805 3789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [term 1 LEADER]: Becoming Leader. State: Replica: 6ca0deef7b3b49bd95c137498893321d, State: Running, Role: LEADER
I20260923 14:45:49.998157 3789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca0deef7b3b49bd95c137498893321d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35505 } }
I20260923 14:45:49.998775 3784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:49.999496 3791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ca0deef7b3b49bd95c137498893321d. Latest consensus state: current_term: 1 leader_uuid: "6ca0deef7b3b49bd95c137498893321d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca0deef7b3b49bd95c137498893321d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35505 } } }
I20260923 14:45:49.999665 3791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:50.000042 3790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ca0deef7b3b49bd95c137498893321d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca0deef7b3b49bd95c137498893321d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35505 } } }
I20260923 14:45:50.000152 3790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:50.000170 3798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:50.003037 3798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:50.008549 3798 catalog_manager.cc:1347] Generated new cluster ID: 4634e33f81fa4ee8924210fe91c63b1d
I20260923 14:45:50.008656 3798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:50.027541 3798 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:50.028527 3798 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:50.033597 3798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ca0deef7b3b49bd95c137498893321d: Generated new TSK 0
I20260923 14:45:50.034281 3798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:45:50.071929 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:50.047203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.30.2.254:35505)
0923 14:45:50.047332 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:50.047362 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:50.047491 (+ 129us) server_negotiation.cc:407] Connection header received
0923 14:45:50.047661 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:50.047664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:50.047721 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:50.047866 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0923 14:45:50.048086 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:50.048908 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:50.049783 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:50.050130 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:50.050231 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:50.050504 (+ 273us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 14:45:50.071382 (+ 20878us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 14:45:50.071736 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40174: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260923 14:45:50.072016 3787 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40174: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 14:45:50.072068 3813 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:45:50.047130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35505 (local address 127.0.0.1:40174)
0923 14:45:50.047409 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:50.047421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:50.047531 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:50.047866 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:50.047868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:50.047875 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 14:45:50.047983 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:50.047989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:50.049021 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:50.049024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:50.049675 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:50.049682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:50.050249 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:50.050252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:50.050319 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:50.050340 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 14:45:50.071644 (+ 21304us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 14:45:50.071889 (+ 245us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.2.254:35505: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
I20260923 14:45:50.072878 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 3717
2026-09-23T14:45:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (292 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:50 dist-test-slave-wjf2 krb5kdc[3825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:50 dist-test-slave-wjf2 krb5kdc[3825](info): set up 2 sockets
Sep 23 14:45:50 dist-test-slave-wjf2 krb5kdc[3825](info): commencing operation
krb5kdc: starting...
W20260923 14:45:51.349893 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:51 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:45:51.470362 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45321
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:45321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:51.582087 3838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:51.582351 3838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:51.582393 3838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:51.586316 3838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:51.586371 3838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:51.586393 3838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:51.586413 3838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:51.586431 3838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:51.591455 3838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:45321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:51.592453 3838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:51.593430 3838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:51.601946 3843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:51.602116 3838 server_base.cc:1034] running on GCE node
W20260923 14:45:51.601935 3846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:51.601938 3844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:51.602771 3838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:51.603765 3838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:51.605036 3838 hybrid_clock.cc:648] HybridClock initialized: now 1790174751605011 us; error 34 us; skew 500 ppm
Sep 23 14:45:51 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174751, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:51.614393 3838 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:45:51.615356 3838 webserver.cc:460] Webserver started at http://127.30.2.254:36199/ using document root <none> and password file <none>
I20260923 14:45:51.616005 3838 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:51.616107 3838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:51.616356 3838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:51.618379 3838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "6ef8e6f6e4cc4222b9a1e66c79c01621"
format_stamp: "Formatted at 2026-09-23 14:45:51 on dist-test-slave-wjf2"
server_key: "555f2e1f43465576f8e7684a3dfc502c"
server_key_iv: "40b4ac86b9aa7876b56ff69d93dcc88c"
server_key_version: "encryptionkey@0"
I20260923 14:45:51.618942 3838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "6ef8e6f6e4cc4222b9a1e66c79c01621"
format_stamp: "Formatted at 2026-09-23 14:45:51 on dist-test-slave-wjf2"
server_key: "555f2e1f43465576f8e7684a3dfc502c"
server_key_iv: "40b4ac86b9aa7876b56ff69d93dcc88c"
server_key_version: "encryptionkey@0"
I20260923 14:45:51.622718 3838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:45:51.625187 3853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:51.626442 3838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:45:51.626578 3838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "6ef8e6f6e4cc4222b9a1e66c79c01621"
format_stamp: "Formatted at 2026-09-23 14:45:51 on dist-test-slave-wjf2"
server_key: "555f2e1f43465576f8e7684a3dfc502c"
server_key_iv: "40b4ac86b9aa7876b56ff69d93dcc88c"
server_key_version: "encryptionkey@0"
I20260923 14:45:51.626685 3838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:51.637447 3838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:51.640326 3838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:51.640522 3838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:51.647511 3838 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:45321
I20260923 14:45:51.647524 3905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:45321 every 8 connection(s)
I20260923 14:45:51.648488 3838 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:51.651355 3906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:51.656071 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 3838
I20260923 14:45:51.656196 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:51.656518 30731 external_mini_cluster.cc:1428] Setting key 7f750435696c7f5cd2cd426017d67a06
I20260923 14:45:51.657708 3906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621: Bootstrap starting.
I20260923 14:45:51.660967 3906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:51.662098 3906 log.cc:824] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:51.664678 3906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621: No bootstrap required, opened a new log
Sep 23 14:45:51 dist-test-slave-wjf2 krb5kdc[3825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:51.668604 3906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8e6f6e4cc4222b9a1e66c79c01621" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45321 } }
I20260923 14:45:51.668825 3906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:51.668893 3906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef8e6f6e4cc4222b9a1e66c79c01621, State: Initialized, Role: FOLLOWER
I20260923 14:45:51.669384 3906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8e6f6e4cc4222b9a1e66c79c01621" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45321 } }
I20260923 14:45:51.669545 3906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:51.669654 3906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:51.669791 3906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:51.670665 3906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8e6f6e4cc4222b9a1e66c79c01621" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45321 } }
I20260923 14:45:51.671101 3906 leader_election.cc:302] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ef8e6f6e4cc4222b9a1e66c79c01621; no voters:
I20260923 14:45:51.671473 3906 leader_election.cc:288] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:51.671582 3911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:51.671815 3911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [term 1 LEADER]: Becoming Leader. State: Replica: 6ef8e6f6e4cc4222b9a1e66c79c01621, State: Running, Role: LEADER
I20260923 14:45:51.672181 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:51.657935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.30.2.254:45321)
0923 14:45:51.658421 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:51.658436 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:51.658470 (+ 34us) server_negotiation.cc:407] Connection header received
0923 14:45:51.659065 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:51.659080 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:51.659385 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:51.659670 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:51.660828 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:51.661832 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:51.662992 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:51.663460 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:51.663575 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:51.666662 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:51.666668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:51.666680 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:51.666706 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:51.669095 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:51.669568 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:51.669574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:51.669580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:51.669746 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:51.670031 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:51.670034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:51.670036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:51.670415 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:51.670618 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:51.670863 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 14:45:51.671172 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":157,"threads_started":1}
I20260923 14:45:51.672144 3911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8e6f6e4cc4222b9a1e66c79c01621" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45321 } }
I20260923 14:45:51.672909 3906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:51.673498 3913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ef8e6f6e4cc4222b9a1e66c79c01621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8e6f6e4cc4222b9a1e66c79c01621" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45321 } } }
I20260923 14:45:51.673609 3913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:51.674494 3916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:51.674979 3912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef8e6f6e4cc4222b9a1e66c79c01621. Latest consensus state: current_term: 1 leader_uuid: "6ef8e6f6e4cc4222b9a1e66c79c01621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef8e6f6e4cc4222b9a1e66c79c01621" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45321 } } }
I20260923 14:45:51.675091 3912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:51.678143 3916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:51.689848 3916 catalog_manager.cc:1347] Generated new cluster ID: dc88372155454f45a22b0dfdf71378bb
I20260923 14:45:51.689924 3916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:51.712327 3916 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:51.713588 3916 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:51.722522 3916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ef8e6f6e4cc4222b9a1e66c79c01621: Generated new TSK 0
I20260923 14:45:51.723320 3916 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:45:51.755832 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45321
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:51.874423 3932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:51.874750 3932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:51.874847 3932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:51.879326 3932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:51.879385 3932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:51.879509 3932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:51.884991 3932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:51.886059 3932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:51.887171 3932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:51.897225 3937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:51.897334 3932 server_base.cc:1034] running on GCE node
W20260923 14:45:51.897234 3938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:51.897240 3940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:51.897914 3932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:51.898538 3932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:51.899719 3932 hybrid_clock.cc:648] HybridClock initialized: now 1790174751899698 us; error 31 us; skew 500 ppm
Sep 23 14:45:51 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174751, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:51.902618 3932 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:45:51.903422 3932 webserver.cc:460] Webserver started at http://127.30.2.193:34835/ using document root <none> and password file <none>
I20260923 14:45:51.904017 3932 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:51.904079 3932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:51.904250 3932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:51.906234 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "c872cf1b8f9f4f4382cf12858e641bf7"
format_stamp: "Formatted at 2026-09-23 14:45:51 on dist-test-slave-wjf2"
server_key: "aa297744f5159b982fb057779e0f11b7"
server_key_iv: "ccc1c183d18cb9818c6fc8a8eb1fad9b"
server_key_version: "encryptionkey@0"
I20260923 14:45:51.906735 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "c872cf1b8f9f4f4382cf12858e641bf7"
format_stamp: "Formatted at 2026-09-23 14:45:51 on dist-test-slave-wjf2"
server_key: "aa297744f5159b982fb057779e0f11b7"
server_key_iv: "ccc1c183d18cb9818c6fc8a8eb1fad9b"
server_key_version: "encryptionkey@0"
I20260923 14:45:51.910483 3932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 14:45:51.912945 3947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:51.914167 3932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:45:51.914261 3932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "c872cf1b8f9f4f4382cf12858e641bf7"
format_stamp: "Formatted at 2026-09-23 14:45:51 on dist-test-slave-wjf2"
server_key: "aa297744f5159b982fb057779e0f11b7"
server_key_iv: "ccc1c183d18cb9818c6fc8a8eb1fad9b"
server_key_version: "encryptionkey@0"
I20260923 14:45:51.914335 3932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:51.935359 3932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:51.938575 3932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:51.938755 3932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:51.939348 3932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:51.940292 3932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:51.940346 3932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:51.940389 3932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:51.940415 3932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:51.949692 3932 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:39555
I20260923 14:45:51.949756 4060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:39555 every 8 connection(s)
I20260923 14:45:51.950593 3932 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:45:51.952514 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 3932
I20260923 14:45:51.952653 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:51.952975 30731 external_mini_cluster.cc:1428] Setting key 80035d6edf3fb1b2059a7d5db4253b9d
Sep 23 14:45:51 dist-test-slave-wjf2 krb5kdc[3825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174751, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
I20260923 14:45:51.965902 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:51.952518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:39575 (local address 127.30.2.254:45321)
0923 14:45:51.952721 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:51.952726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:51.953614 (+ 888us) server_negotiation.cc:407] Connection header received
0923 14:45:51.955011 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:51.955018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:51.955082 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:51.955176 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:51.956422 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:51.957275 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:51.959071 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:51.959295 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:51.959407 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:51.962223 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:51.962226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:51.962231 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:51.962260 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:51.963703 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:51.964249 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:51.964254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:51.964256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:51.964336 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:51.964695 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:51.964698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:51.964705 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:51.964923 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:51.965095 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:51.965601 (+ 506us) server_negotiation.cc:299] Negotiation successful
0923 14:45:51.965724 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 14:45:51.966683 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:51.952953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45321 (local address 127.30.2.193:39575)
0923 14:45:51.953456 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:51.953508 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:51.954800 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:51.955329 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:51.955336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:51.955677 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:51.956268 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:51.956283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:51.957440 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:51.957446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:51.958951 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:51.958966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:51.959480 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:51.959487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:51.959711 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:51.960399 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:51.960426 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:51.962056 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:51.963830 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:51.963835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:51.963850 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:51.964144 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:51.964441 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:51.964444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:51.964445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:51.964579 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:51.965095 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:51.965104 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:51.965347 (+ 243us) client_negotiation.cc:769] Sending connection context
0923 14:45:51.965542 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 14:45:51.965869 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":145,"threads_started":1}
I20260923 14:45:51.967931 4061 heartbeater.cc:347] Connected to a master server at 127.30.2.254:45321
I20260923 14:45:51.968204 4061 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:51.968688 4061 heartbeater.cc:511] Master 127.30.2.254:45321 requested a full tablet report, sending...
I20260923 14:45:51.970207 3870 ts_manager.cc:194] Registered new tserver with Master: c872cf1b8f9f4f4382cf12858e641bf7 (127.30.2.193:39555)
I20260923 14:45:51.971328 3870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:39575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:45:51.984162 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45321
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:52.094795 4066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:52.095048 4066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:52.095153 4066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:52.099274 4066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:52.099344 4066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:52.099423 4066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:52.104568 4066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:52.105546 4066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:52.106540 4066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:52.115815 4074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:52.115830 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:52.115862 4066 server_base.cc:1034] running on GCE node
W20260923 14:45:52.115775 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:52.116624 4066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:52.117247 4066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:52.118433 4066 hybrid_clock.cc:648] HybridClock initialized: now 1790174752118413 us; error 27 us; skew 500 ppm
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:52.121133 4066 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:45:52.121961 4066 webserver.cc:460] Webserver started at http://127.30.2.194:33183/ using document root <none> and password file <none>
I20260923 14:45:52.122500 4066 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:52.122586 4066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:52.122816 4066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:52.124835 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "bb4778118c2b4c71996ef5114539fc81"
format_stamp: "Formatted at 2026-09-23 14:45:52 on dist-test-slave-wjf2"
server_key: "837d6f4edf70c72cbb6a7b052b2fa698"
server_key_iv: "c534432bb9eb82e6f6c8345400482bb7"
server_key_version: "encryptionkey@0"
I20260923 14:45:52.125379 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "bb4778118c2b4c71996ef5114539fc81"
format_stamp: "Formatted at 2026-09-23 14:45:52 on dist-test-slave-wjf2"
server_key: "837d6f4edf70c72cbb6a7b052b2fa698"
server_key_iv: "c534432bb9eb82e6f6c8345400482bb7"
server_key_version: "encryptionkey@0"
I20260923 14:45:52.128922 4066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:45:52.131492 4081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:52.132745 4066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:52.132871 4066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "bb4778118c2b4c71996ef5114539fc81"
format_stamp: "Formatted at 2026-09-23 14:45:52 on dist-test-slave-wjf2"
server_key: "837d6f4edf70c72cbb6a7b052b2fa698"
server_key_iv: "c534432bb9eb82e6f6c8345400482bb7"
server_key_version: "encryptionkey@0"
I20260923 14:45:52.132980 4066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:52.149088 4066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:52.151901 4066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:52.152091 4066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:52.152657 4066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:52.153450 4066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:52.153515 4066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:52.153584 4066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:52.153622 4066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:52.162294 4066 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:40253
I20260923 14:45:52.162330 4194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:40253 every 8 connection(s)
I20260923 14:45:52.163161 4066 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:52.170226 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 4066
I20260923 14:45:52.170351 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:52.170682 30731 external_mini_cluster.cc:1428] Setting key a9574564f55aed069140512f01058cb2
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:52.177728 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:52.164978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:36427 (local address 127.30.2.254:45321)
0923 14:45:52.165140 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:52.165145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:52.165944 (+ 799us) server_negotiation.cc:407] Connection header received
0923 14:45:52.167256 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:52.167259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:52.167306 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:52.167380 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:52.168542 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.169053 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.170238 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.170581 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.170707 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:52.173306 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:52.173332 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:52.173336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:52.173372 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:52.175403 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:52.176030 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:52.176032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:52.176034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:52.176116 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:52.176525 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:52.176527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:52.176528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:52.176695 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:52.176767 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:52.177487 (+ 720us) server_negotiation.cc:299] Negotiation successful
0923 14:45:52.177604 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 14:45:52.178944 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:52.165359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45321 (local address 127.30.2.194:36427)
0923 14:45:52.165800 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:52.165840 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:52.167062 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:52.167541 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:52.167547 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:52.167964 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:52.168419 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:52.168431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.169193 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.169196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:52.170104 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:52.170115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.170750 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.170757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:52.170914 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:52.171643 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:52.171659 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:52.173149 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:52.175531 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:52.175539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:52.175554 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:52.175935 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:52.176231 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:52.176237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:52.176242 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:52.176426 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:52.176803 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:52.176813 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:52.177144 (+ 331us) client_negotiation.cc:769] Sending connection context
0923 14:45:52.177399 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 14:45:52.177754 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
I20260923 14:45:52.180264 4195 heartbeater.cc:347] Connected to a master server at 127.30.2.254:45321
I20260923 14:45:52.180521 4195 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:52.181020 4195 heartbeater.cc:511] Master 127.30.2.254:45321 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:45:52.182173 3870 ts_manager.cc:194] Registered new tserver with Master: bb4778118c2b4c71996ef5114539fc81 (127.30.2.194:40253)
I20260923 14:45:52.182824 3870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:36427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:45:52.199235 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45321
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:52.313647 4200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:52.313942 4200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:52.314013 4200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:52.318244 4200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:52.318326 4200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:52.318416 4200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:52.323869 4200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:52.324975 4200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:52.326040 4200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:52.335940 4205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:52.335958 4206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:52.336026 4200 server_base.cc:1034] running on GCE node
W20260923 14:45:52.335958 4208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:52.336692 4200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:52.337333 4200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:52.338501 4200 hybrid_clock.cc:648] HybridClock initialized: now 1790174752338485 us; error 27 us; skew 500 ppm
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:52.341233 4200 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:45:52.342018 4200 webserver.cc:460] Webserver started at http://127.30.2.195:41175/ using document root <none> and password file <none>
I20260923 14:45:52.342532 4200 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:52.342585 4200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:52.342756 4200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:52.344836 4200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "a17820408e294fd48462e346c8a7b4ac"
format_stamp: "Formatted at 2026-09-23 14:45:52 on dist-test-slave-wjf2"
server_key: "3e133f6a0c704650f3f7d5f4478a52e5"
server_key_iv: "fe15a861daa9c42d59656081de73621f"
server_key_version: "encryptionkey@0"
I20260923 14:45:52.345342 4200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "a17820408e294fd48462e346c8a7b4ac"
format_stamp: "Formatted at 2026-09-23 14:45:52 on dist-test-slave-wjf2"
server_key: "3e133f6a0c704650f3f7d5f4478a52e5"
server_key_iv: "fe15a861daa9c42d59656081de73621f"
server_key_version: "encryptionkey@0"
I20260923 14:45:52.349171 4200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:45:52.351501 4215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:52.352762 4200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:52.352892 4200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "a17820408e294fd48462e346c8a7b4ac"
format_stamp: "Formatted at 2026-09-23 14:45:52 on dist-test-slave-wjf2"
server_key: "3e133f6a0c704650f3f7d5f4478a52e5"
server_key_iv: "fe15a861daa9c42d59656081de73621f"
server_key_version: "encryptionkey@0"
I20260923 14:45:52.352999 4200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:52.366555 4200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:52.369380 4200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:52.369578 4200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:52.370191 4200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:52.371001 4200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:52.371069 4200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:52.371146 4200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:52.371191 4200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:52.380182 4200 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:34469
I20260923 14:45:52.380219 4328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:34469 every 8 connection(s)
I20260923 14:45:52.381054 4200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:45:52.386168 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 4200
I20260923 14:45:52.386313 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:45:52.386682 30731 external_mini_cluster.cc:1428] Setting key 14391540265a6c7ad9ddffde6da078cf
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:45:52.394605 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:52.382879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:41895 (local address 127.30.2.254:45321)
0923 14:45:52.383021 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:52.383024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:52.383823 (+ 799us) server_negotiation.cc:407] Connection header received
0923 14:45:52.385180 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:52.385182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:52.385230 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:52.385304 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:52.386519 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.387246 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.388540 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.388777 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.388893 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:52.391302 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:52.391306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:52.391308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:52.391339 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:52.392625 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:52.393144 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:52.393150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:52.393152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:52.393232 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:52.393567 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:52.393572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:52.393573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:52.393755 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:52.393862 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:52.394328 (+ 466us) server_negotiation.cc:299] Negotiation successful
0923 14:45:52.394439 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:45:52.395346 4331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:52.383240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45321 (local address 127.30.2.195:41895)
0923 14:45:52.383654 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:52.383695 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:52.384982 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:52.385464 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:52.385471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:52.385795 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:52.386321 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:52.386344 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.387486 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.387489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:52.388410 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:52.388420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.388940 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.388944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:52.389106 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:52.389724 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:52.389739 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:52.391093 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:52.392755 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:52.392761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:52.392774 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:52.393041 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:52.393334 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:52.393337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:52.393339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:52.393471 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:52.393860 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:52.393867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:52.394095 (+ 228us) client_negotiation.cc:769] Sending connection context
0923 14:45:52.394286 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 14:45:52.394573 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260923 14:45:52.396378 4329 heartbeater.cc:347] Connected to a master server at 127.30.2.254:45321
I20260923 14:45:52.396586 4329 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:52.397018 4329 heartbeater.cc:511] Master 127.30.2.254:45321 requested a full tablet report, sending...
I20260923 14:45:52.397986 3870 ts_manager.cc:194] Registered new tserver with Master: a17820408e294fd48462e346c8a7b4ac (127.30.2.195:34469)
I20260923 14:45:52.398487 3870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:41895
I20260923 14:45:52.400404 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/test-admin.keytab.
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:52.416186 30731 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 14:45:52.418895 4336 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:45:52.417485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.30.2.193:39555)
0923 14:45:52.417771 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:52.417819 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:52.417848 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:45:52.417925 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:52.417933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:52.418139 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:52.418258 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:52.418709 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36964: BlockingRecv error: recv got EOF from 127.0.0.1:36964 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
W20260923 14:45:52.418887 30731 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/krb5cc' of type FILE
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:52.419937 30731 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:52.420888 30731 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:45:52.430384 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:52.421726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.30.2.193:39555)
0923 14:45:52.421864 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:52.421869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:52.421977 (+ 108us) server_negotiation.cc:407] Connection header received
0923 14:45:52.422116 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:52.422119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:52.422165 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:52.422250 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:52.422768 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.423536 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.424425 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.424801 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.424919 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:52.427024 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:52.427033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:52.427045 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:52.427067 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:52.428600 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:52.429035 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:52.429041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:52.429042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:52.429117 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:52.429397 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:52.429401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:52.429403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:52.429632 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:52.429846 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:52.430051 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 14:45:52.430200 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 14:45:52.434301 4336 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:45:52.433294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36970 (local address 127.30.2.193:39555)
0923 14:45:52.433412 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:52.433416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:52.433519 (+ 103us) server_negotiation.cc:407] Connection header received
0923 14:45:52.433646 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:52.433648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:52.433692 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:52.433773 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:52.434192 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36970: BlockingRecv error: recv got EOF from 127.0.0.1:36970 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 14:45:52.434420 30731 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/krb5cc' of type FILE
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:52.435353 30731 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:52.436288 30731 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:45:52.444124 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:52.437076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.30.2.193:39555)
0923 14:45:52.437198 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:52.437202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:52.437362 (+ 160us) server_negotiation.cc:407] Connection header received
0923 14:45:52.437431 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:52.437433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:52.437464 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:52.437548 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:52.437977 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.438475 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.439178 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.439383 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.439456 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:52.441436 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:52.441439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:52.441440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:52.441460 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:52.442664 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:52.443027 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:52.443030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:52.443031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:52.443101 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:52.443319 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:52.443321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:52.443322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:52.443461 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:52.443552 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:52.443769 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 14:45:52.443960 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260923 14:45:52.446513 4336 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:45:52.445678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36984 (local address 127.30.2.193:39555)
0923 14:45:52.445780 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:52.445783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:52.445953 (+ 170us) server_negotiation.cc:407] Connection header received
0923 14:45:52.446019 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:52.446022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:52.446054 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:52.446113 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:52.446419 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36984: BlockingRecv error: recv got EOF from 127.0.0.1:36984 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 14:45:52.446628 30731 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174723338690-30731-0/krb5kdc/krb5cc' of type FILE
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:52.447540 30731 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:52.448421 30731 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[3825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:45:52.456195 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:52.449187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36994 (local address 127.30.2.193:39555)
0923 14:45:52.449292 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:52.449296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:52.449473 (+ 177us) server_negotiation.cc:407] Connection header received
0923 14:45:52.449541 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:52.449543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:52.449575 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:52.449641 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:52.450058 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.450539 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.451259 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:52.451454 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:52.451526 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:52.453641 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:52.453644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:52.453646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:52.453666 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:52.454844 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:52.455197 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:52.455199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:52.455200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:52.455267 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:52.455488 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:52.455490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:52.455491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:52.455636 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:52.455729 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:52.455966 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 14:45:52.456068 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:45:52.457139 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 3932
I20260923 14:45:52.463804 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 4066
I20260923 14:45:52.469517 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 4200
I20260923 14:45:52.475035 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 3838
2026-09-23T14:45:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2159 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[4349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[4349](info): set up 2 sockets
Sep 23 14:45:52 dist-test-slave-wjf2 krb5kdc[4349](info): commencing operation
krb5kdc: starting...
W20260923 14:45:53.537741 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:53 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.254.keytab.
Entry for principal oryx/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.254.keytab.
I20260923 14:45:53.659083 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:32951
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.254.keytab
--principal=oryx/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:32951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:53.767100 4362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:53.767402 4362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:53.767474 4362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:53.771431 4362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:53.771516 4362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:53.771555 4362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:53.771615 4362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:53.771649 4362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:53.776784 4362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:32951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.254.keytab
--principal=oryx/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:32951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:53.777837 4362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:53.778949 4362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:53.787633 4370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:53.787649 4367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:53.787684 4368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:53.787760 4362 server_base.cc:1034] running on GCE node
I20260923 14:45:53.788561 4362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:53.789681 4362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:53.790854 4362 hybrid_clock.cc:648] HybridClock initialized: now 1790174753790809 us; error 55 us; skew 500 ppm
Sep 23 14:45:53 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:53.793649 4362 init.cc:377] Logged in from keytab as oryx/127.30.2.254@KRBTEST.COM (short username oryx)
I20260923 14:45:53.794569 4362 webserver.cc:460] Webserver started at http://127.30.2.254:42597/ using document root <none> and password file <none>
I20260923 14:45:53.795145 4362 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:53.795235 4362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:53.795467 4362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:53.797550 4362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "6ba2e576559541c8b9fe54558d267710"
format_stamp: "Formatted at 2026-09-23 14:45:53 on dist-test-slave-wjf2"
server_key: "092713f89aad15d5ab3547e8d0c45779"
server_key_iv: "8df42710be52b5f89580ae96bc223796"
server_key_version: "encryptionkey@0"
I20260923 14:45:53.798128 4362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "6ba2e576559541c8b9fe54558d267710"
format_stamp: "Formatted at 2026-09-23 14:45:53 on dist-test-slave-wjf2"
server_key: "092713f89aad15d5ab3547e8d0c45779"
server_key_iv: "8df42710be52b5f89580ae96bc223796"
server_key_version: "encryptionkey@0"
I20260923 14:45:53.801824 4362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:45:53.804282 4377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:53.805526 4362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:53.805666 4362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "6ba2e576559541c8b9fe54558d267710"
format_stamp: "Formatted at 2026-09-23 14:45:53 on dist-test-slave-wjf2"
server_key: "092713f89aad15d5ab3547e8d0c45779"
server_key_iv: "8df42710be52b5f89580ae96bc223796"
server_key_version: "encryptionkey@0"
I20260923 14:45:53.805770 4362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:53.821199 4362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:53.824126 4362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:53.824319 4362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:53.831243 4362 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:32951
I20260923 14:45:53.831527 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:32951 every 8 connection(s)
I20260923 14:45:53.832264 4362 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:53.834450 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 4362
I20260923 14:45:53.834589 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:53.834905 30731 external_mini_cluster.cc:1428] Setting key 230d39d2b0873fff811f6dc2faee7d53
I20260923 14:45:53.835356 4430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:53.842453 4430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710: Bootstrap starting.
Sep 23 14:45:53 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:45:53.845103 4430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:53.846021 4430 log.cc:824] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:53.848280 4430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710: No bootstrap required, opened a new log
I20260923 14:45:53.850214 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:53.836343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.30.2.254:32951)
0923 14:45:53.836788 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:53.836809 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:53.836851 (+ 42us) server_negotiation.cc:407] Connection header received
0923 14:45:53.837661 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:53.837684 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:53.838086 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:53.838443 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:53.839277 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:53.840831 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:53.841644 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:53.842350 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:53.842528 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:53.845342 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:53.845350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:53.845369 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:53.845405 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:53.847289 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:53.847698 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:53.847704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:53.847709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:53.847861 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:53.848126 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:53.848129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:53.848131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:53.848618 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:53.848809 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:53.849011 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 14:45:53.849291 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":129,"threads_started":1}
I20260923 14:45:53.852382 4430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } }
I20260923 14:45:53.852595 4430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:53.852638 4430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba2e576559541c8b9fe54558d267710, State: Initialized, Role: FOLLOWER
I20260923 14:45:53.853017 4430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } }
I20260923 14:45:53.853125 4430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:53.853164 4430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:53.853235 4430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:53.854079 4430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } }
I20260923 14:45:53.854429 4430 leader_election.cc:302] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ba2e576559541c8b9fe54558d267710; no voters:
I20260923 14:45:53.854782 4430 leader_election.cc:288] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:53.854856 4435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:53.855159 4435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 1 LEADER]: Becoming Leader. State: Replica: 6ba2e576559541c8b9fe54558d267710, State: Running, Role: LEADER
I20260923 14:45:53.855532 4435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } }
I20260923 14:45:53.856015 4430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:53.857368 4436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } }
I20260923 14:45:53.857378 4437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba2e576559541c8b9fe54558d267710. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } }
I20260923 14:45:53.857503 4436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:53.857517 4437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:53.857899 4445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:53.860958 4445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:53.866466 4445 catalog_manager.cc:1347] Generated new cluster ID: ea6a320ed7bf45599adce879d795567e
I20260923 14:45:53.866551 4445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:53.874497 4445 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:53.875447 4445 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:53.881932 4445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710: Generated new TSK 0
I20260923 14:45:53.882800 4445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab.
Entry for principal oryx/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab.
I20260923 14:45:53.916930 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:32951
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab
--principal=oryx/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:54.030112 4456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:54.030377 4456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:54.030421 4456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:54.034574 4456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:54.034627 4456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:54.034718 4456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:54.040037 4456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab
--principal=oryx/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:32951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:54.041042 4456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:54.042106 4456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:54.051523 4461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:54.051630 4462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:54.051604 4464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:54.051851 4456 server_base.cc:1034] running on GCE node
I20260923 14:45:54.052230 4456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:54.052870 4456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:54.054054 4456 hybrid_clock.cc:648] HybridClock initialized: now 1790174754054038 us; error 33 us; skew 500 ppm
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:54.056880 4456 init.cc:377] Logged in from keytab as oryx/127.30.2.193@KRBTEST.COM (short username oryx)
I20260923 14:45:54.057825 4456 webserver.cc:460] Webserver started at http://127.30.2.193:41143/ using document root <none> and password file <none>
I20260923 14:45:54.058423 4456 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:54.058513 4456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:54.058743 4456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:54.060803 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "6c974d96d963445caf911d5aee86d92a"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "cbd94cb4b3d42f6b14cbfbb5815efc44"
server_key_iv: "e0d8072ae4b65adc2cb79868e0c13f4c"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.061350 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "6c974d96d963445caf911d5aee86d92a"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "cbd94cb4b3d42f6b14cbfbb5815efc44"
server_key_iv: "e0d8072ae4b65adc2cb79868e0c13f4c"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.065029 4456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:45:54.067400 4471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.068681 4456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:45:54.068823 4456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "6c974d96d963445caf911d5aee86d92a"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "cbd94cb4b3d42f6b14cbfbb5815efc44"
server_key_iv: "e0d8072ae4b65adc2cb79868e0c13f4c"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.068933 4456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:54.088297 4456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:54.091017 4456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:54.091240 4456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:54.091864 4456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:54.092657 4456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:54.092737 4456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.092805 4456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:54.092855 4456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.102197 4456 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:42417
I20260923 14:45:54.102223 4584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:42417 every 8 connection(s)
I20260923 14:45:54.103045 4456 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:45:54.112977 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 4456
I20260923 14:45:54.113119 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:54.113499 30731 external_mini_cluster.cc:1428] Setting key e1f3669e99fe05413ee1d19fab74d66e
I20260923 14:45:54.117033 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.104863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:49351 (local address 127.30.2.254:32951)
0923 14:45:54.105027 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:54.105031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:54.105845 (+ 814us) server_negotiation.cc:407] Connection header received
0923 14:45:54.107177 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:54.107179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:54.107226 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:54.107299 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:54.108444 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.108996 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.110205 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.110435 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.110519 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.112968 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:54.112972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:54.112975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:54.113005 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:54.114679 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.115265 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.115272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.115275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.115364 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.115752 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.115758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.115763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.116038 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:54.116166 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:54.116706 (+ 540us) server_negotiation.cc:299] Negotiation successful
0923 14:45:54.116845 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 14:45:54.117805 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.105265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.193:49351)
0923 14:45:54.105701 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:54.105742 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:54.106957 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:54.107456 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:54.107462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:54.107864 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:54.108313 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.108327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.109089 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.109092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.110100 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.110110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.110562 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.110565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.110712 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.111237 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:54.111253 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:54.112799 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:54.114832 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.114838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.114851 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.115163 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.115488 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.115491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.115493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.115639 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.116181 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:54.116186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:54.116398 (+ 212us) client_negotiation.cc:769] Sending connection context
0923 14:45:54.116624 (+ 226us) client_negotiation.cc:240] Negotiation successful
0923 14:45:54.116955 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":115,"threads_started":1}
I20260923 14:45:54.119160 4585 heartbeater.cc:347] Connected to a master server at 127.30.2.254:32951
I20260923 14:45:54.119407 4585 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:54.119931 4585 heartbeater.cc:511] Master 127.30.2.254:32951 requested a full tablet report, sending...
I20260923 14:45:54.121474 4394 ts_manager.cc:194] Registered new tserver with Master: 6c974d96d963445caf911d5aee86d92a (127.30.2.193:42417)
I20260923 14:45:54.122565 4394 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:49351
WARNING: no policy specified for oryx/127.30.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab.
Entry for principal oryx/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab.
I20260923 14:45:54.142288 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:32951
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab
--principal=oryx/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:54.250895 4590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:54.251128 4590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:54.251199 4590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:54.255223 4590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:54.255276 4590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:54.255374 4590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:54.261022 4590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab
--principal=oryx/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:32951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:54.262022 4590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:54.263038 4590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:54.272773 4596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:54.272930 4590 server_base.cc:1034] running on GCE node
W20260923 14:45:54.272755 4595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:54.272747 4598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:54.273528 4590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:54.274159 4590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:54.275319 4590 hybrid_clock.cc:648] HybridClock initialized: now 1790174754275307 us; error 35 us; skew 500 ppm
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:54.278152 4590 init.cc:377] Logged in from keytab as oryx/127.30.2.194@KRBTEST.COM (short username oryx)
I20260923 14:45:54.279030 4590 webserver.cc:460] Webserver started at http://127.30.2.194:36453/ using document root <none> and password file <none>
I20260923 14:45:54.279646 4590 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:54.279740 4590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:54.280002 4590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:54.282143 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "1f02345dda9e49928bda61615714a6b8"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "456a46d39f9e444c7b1ab7e4e23c95a0"
server_key_iv: "d1471d63b7e9fdc48a93849cc718edc0"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.282727 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "1f02345dda9e49928bda61615714a6b8"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "456a46d39f9e444c7b1ab7e4e23c95a0"
server_key_iv: "d1471d63b7e9fdc48a93849cc718edc0"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.286700 4590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 14:45:54.289230 4605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.290465 4590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:45:54.290598 4590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "1f02345dda9e49928bda61615714a6b8"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "456a46d39f9e444c7b1ab7e4e23c95a0"
server_key_iv: "d1471d63b7e9fdc48a93849cc718edc0"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.290726 4590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:54.319249 4590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:54.322261 4590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:54.322461 4590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:54.323056 4590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:54.323987 4590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:54.324054 4590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.324131 4590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:54.324174 4590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.333370 4590 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:41339
I20260923 14:45:54.333415 4718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:41339 every 8 connection(s)
I20260923 14:45:54.334272 4590 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:54.338099 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 4590
I20260923 14:45:54.338196 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:54.338513 30731 external_mini_cluster.cc:1428] Setting key 6f406cf9b5b46e6651309dcec816bf8a
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:45:54.348536 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.336092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:53609 (local address 127.30.2.254:32951)
0923 14:45:54.336282 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:54.336287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:54.337084 (+ 797us) server_negotiation.cc:407] Connection header received
0923 14:45:54.338587 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:54.338591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:54.338647 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:54.338728 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:54.340039 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.340574 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.341699 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.341927 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.342039 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.344806 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:54.344809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:54.344811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:54.344834 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:54.346237 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.346899 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.346902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.346904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.347000 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.347394 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.347396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.347397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.347559 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:54.347634 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:54.348313 (+ 679us) server_negotiation.cc:299] Negotiation successful
0923 14:45:54.348416 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 14:45:54.349501 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.336507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.194:53609)
0923 14:45:54.336941 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:54.336981 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:54.338374 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:54.338919 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:54.338928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:54.339320 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:54.339878 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.339900 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.340702 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.340706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.341564 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.341573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.342076 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.342081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.342250 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.342845 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:54.342861 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:54.344631 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:54.346356 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.346364 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.346380 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.346735 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.347100 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.347104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.347108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.347294 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.347672 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:54.347682 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:54.348029 (+ 347us) client_negotiation.cc:769] Sending connection context
0923 14:45:54.348252 (+ 223us) client_negotiation.cc:240] Negotiation successful
0923 14:45:54.348587 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260923 14:45:54.350741 4719 heartbeater.cc:347] Connected to a master server at 127.30.2.254:32951
I20260923 14:45:54.351007 4719 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.195@KRBTEST.COM" created.
I20260923 14:45:54.351495 4719 heartbeater.cc:511] Master 127.30.2.254:32951 requested a full tablet report, sending...
I20260923 14:45:54.352802 4394 ts_manager.cc:194] Registered new tserver with Master: 1f02345dda9e49928bda61615714a6b8 (127.30.2.194:41339)
I20260923 14:45:54.353374 4394 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:53609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab.
Entry for principal oryx/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab.
I20260923 14:45:54.368127 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:32951
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab
--principal=oryx/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:54.478669 4724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:54.478884 4724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:54.478922 4724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:54.483078 4724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:54.483126 4724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:54.483208 4724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:54.488260 4724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab
--principal=oryx/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:32951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:54.489215 4724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:54.490183 4724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:54.499766 4729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:54.499926 4724 server_base.cc:1034] running on GCE node
W20260923 14:45:54.499766 4732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:54.499763 4730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:54.500533 4724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:54.501138 4724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:54.502337 4724 hybrid_clock.cc:648] HybridClock initialized: now 1790174754502304 us; error 45 us; skew 500 ppm
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:54.505144 4724 init.cc:377] Logged in from keytab as oryx/127.30.2.195@KRBTEST.COM (short username oryx)
I20260923 14:45:54.505996 4724 webserver.cc:460] Webserver started at http://127.30.2.195:39433/ using document root <none> and password file <none>
I20260923 14:45:54.506500 4724 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:54.506557 4724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:54.506718 4724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:54.508860 4724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "5d3f26ce1e3a472281fbd8391587db1f"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "6c26efd529fa8cf2858e5c7a549d7ffd"
server_key_iv: "0494184373a2bf55df13667c20e3b88f"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.509442 4724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "5d3f26ce1e3a472281fbd8391587db1f"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "6c26efd529fa8cf2858e5c7a549d7ffd"
server_key_iv: "0494184373a2bf55df13667c20e3b88f"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.513275 4724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:45:54.515664 4739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.516984 4724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:54.517105 4724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "5d3f26ce1e3a472281fbd8391587db1f"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "6c26efd529fa8cf2858e5c7a549d7ffd"
server_key_iv: "0494184373a2bf55df13667c20e3b88f"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.517194 4724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:54.531498 4724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:54.534754 4724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:54.534905 4724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:54.535423 4724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:54.536281 4724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:54.536327 4724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.536366 4724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:54.536387 4724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.545497 4724 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:33897
I20260923 14:45:54.545535 4852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:33897 every 8 connection(s)
I20260923 14:45:54.546393 4724 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:45:54.553869 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 4724
I20260923 14:45:54.554011 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:45:54.554293 30731 external_mini_cluster.cc:1428] Setting key 460cc5ff03d0a6d8afa476507eb755d7
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:45:54.559935 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.548263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:34165 (local address 127.30.2.254:32951)
0923 14:45:54.548407 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:54.548411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:54.549283 (+ 872us) server_negotiation.cc:407] Connection header received
0923 14:45:54.550671 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:54.550675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:54.550745 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:54.550848 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:54.552036 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.552571 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.553707 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.553972 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.554094 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.556784 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:54.556788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:54.556791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:54.556822 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:54.558166 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.558634 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.558636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.558637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.558708 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.558993 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.558995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.558996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.559149 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:54.559221 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:54.559671 (+ 450us) server_negotiation.cc:299] Negotiation successful
0923 14:45:54.559767 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:45:54.560758 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.548690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.195:34165)
0923 14:45:54.549127 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:54.549168 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:54.550452 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:54.551042 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:54.551049 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:54.551403 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:54.551887 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.551900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.552699 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.552703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.553576 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.553586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.554129 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.554133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.554311 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.554945 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:54.554964 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:54.556619 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:54.558285 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.558289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.558300 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.558547 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.558793 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.558795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.558797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.558916 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.559233 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:54.559238 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:54.559479 (+ 241us) client_negotiation.cc:769] Sending connection context
0923 14:45:54.559629 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 14:45:54.559931 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260923 14:45:54.561882 4853 heartbeater.cc:347] Connected to a master server at 127.30.2.254:32951
I20260923 14:45:54.562136 4853 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:54.562634 4853 heartbeater.cc:511] Master 127.30.2.254:32951 requested a full tablet report, sending...
I20260923 14:45:54.563648 4394 ts_manager.cc:194] Registered new tserver with Master: 5d3f26ce1e3a472281fbd8391587db1f (127.30.2.195:33897)
I20260923 14:45:54.564296 4394 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:34165
I20260923 14:45:54.568547 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.2.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.253.keytab.
Entry for principal oryx/127.30.2.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.253.keytab.
I20260923 14:45:54.595520 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.253:43049
--webserver_interface=127.30.2.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.253.keytab
--principal=oryx/127.30.2.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:32951,127.30.2.253:43049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:54.717233 4858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:54.717531 4858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:54.717620 4858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:54.721935 4858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:54.722012 4858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:54.722048 4858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:54.722113 4858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:54.722150 4858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:54.727484 4858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:32951,127.30.2.253:43049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.253.keytab
--principal=oryx/127.30.2.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.253:43049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.2.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
W20260923 14:45:54.727741 4858 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.30.2.254:32951,127.30.2.253:43049'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 14:45:54.728648 4858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:54.729694 4858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:54.738649 4863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:54.738656 4866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:54.738770 4858 server_base.cc:1034] running on GCE node
W20260923 14:45:54.738969 4864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:54.739486 4858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:54.740680 4858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:54.741873 4858 hybrid_clock.cc:648] HybridClock initialized: now 1790174754741842 us; error 49 us; skew 500 ppm
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:54.744724 4858 init.cc:377] Logged in from keytab as oryx/127.30.2.253@KRBTEST.COM (short username oryx)
I20260923 14:45:54.745720 4858 webserver.cc:460] Webserver started at http://127.30.2.253:33257/ using document root <none> and password file <none>
I20260923 14:45:54.746349 4858 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:54.746452 4858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:54.746724 4858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:54.749146 4858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data/instance:
uuid: "d393dc58431a42f98fc87bdf91dff6c3"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "7c1db8f3a2a968194ea43a24c82cb3b8"
server_key_iv: "aa6154b123137db424e7fda031f82c06"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.749788 4858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/wal/instance:
uuid: "d393dc58431a42f98fc87bdf91dff6c3"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "7c1db8f3a2a968194ea43a24c82cb3b8"
server_key_iv: "aa6154b123137db424e7fda031f82c06"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.753834 4858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:45:54.756613 4873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.758116 4858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:45:54.758214 4858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/wal
uuid: "d393dc58431a42f98fc87bdf91dff6c3"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "7c1db8f3a2a968194ea43a24c82cb3b8"
server_key_iv: "aa6154b123137db424e7fda031f82c06"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.758292 4858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:54.768061 4858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:54.770927 4858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:54.771075 4858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:54.778312 4858 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.253:43049
I20260923 14:45:54.778596 4925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.253:43049 every 8 connection(s)
I20260923 14:45:54.779299 4858 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
I20260923 14:45:54.782171 4926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:54.782852 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 4858
I20260923 14:45:54.786417 4926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.2.253@KRBTEST.COM
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.253@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:45:54.798561 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.788205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.30.2.254:32951)
0923 14:45:54.788348 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:54.788353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:54.788705 (+ 352us) server_negotiation.cc:407] Connection header received
0923 14:45:54.788934 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:54.788937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:54.788985 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:54.789050 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:54.789891 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.790650 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.791927 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.792178 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.792285 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.794854 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:54.794861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:54.794865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:54.794904 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:54.796794 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.797266 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.797271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.797273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.797354 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.797629 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.797640 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.797641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.797847 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:54.797955 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:54.798290 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 14:45:54.798398 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:45:54.799243 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.788142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:34862)
0923 14:45:54.788533 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:54.788574 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:54.788806 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:54.789132 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:54.789138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:54.789559 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:54.789741 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.789751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.790775 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.790778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.791764 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.791796 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.792353 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.792360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.792495 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.793058 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:54.793079 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:54.794648 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:54.796916 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.796924 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.796928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.797128 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.797455 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.797458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.797460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.797538 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.797967 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:54.797975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:54.798081 (+ 106us) client_negotiation.cc:769] Sending connection context
0923 14:45:54.798222 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 14:45:54.798400 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":149,"threads_started":1}
I20260923 14:45:54.799250 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.784427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.30.2.253:43049)
0923 14:45:54.784847 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:54.784862 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:54.784898 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:45:54.785670 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:54.785690 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:54.786034 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:54.786383 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:54.787230 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.788534 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.789772 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.790518 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.790683 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.793657 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:54.793664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:54.793674 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:54.793700 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:54.795736 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.796229 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.796239 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.796246 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.796398 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.796696 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.796702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.796705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.797327 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:54.797603 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:54.797943 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 14:45:54.798287 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260923 14:45:54.800674 4926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.253@KRBTEST.COM for oryx/127.30.2.253@KRBTEST.COM
I20260923 14:45:54.809376 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.801235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.30.2.253:43049)
0923 14:45:54.801366 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:54.801372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:54.801392 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:45:54.801492 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:54.801496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:54.801543 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:54.801770 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:54.802456 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.803005 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.803750 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.804231 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.804403 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.806382 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:54.806386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:54.806388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:54.806411 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:54.807827 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.808273 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.808276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.808277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.808350 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.808665 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.808671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.808675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.808839 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:54.808914 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:54.809170 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 14:45:54.809256 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:45:54.809353 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.801128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43049 (local address 127.0.0.1:45654)
0923 14:45:54.801313 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:54.801325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:54.801411 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:54.801789 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:54.801791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:54.801911 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:54.802335 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.802342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.803150 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.803153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.803628 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.803636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.804399 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.804402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.804483 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.804978 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:54.804991 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:54.806253 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:54.807950 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.807952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.807953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.808163 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.808460 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.808462 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.808464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.808547 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.808924 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:54.808926 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:54.808998 (+ 72us) client_negotiation.cc:769] Sending connection context
0923 14:45:54.809121 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 14:45:54.809209 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260923 14:45:54.812420 4926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: Bootstrap starting.
I20260923 14:45:54.814738 4926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:54.815459 4926 log.cc:824] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:54.817351 4926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: No bootstrap required, opened a new log
I20260923 14:45:54.820322 4926 raft_consensus.cc:348] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } }
I20260923 14:45:54.820566 4926 raft_consensus.cc:374] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:54.820637 4926 raft_consensus.cc:729] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d393dc58431a42f98fc87bdf91dff6c3, State: Initialized, Role: FOLLOWER
I20260923 14:45:54.821097 4926 consensus_queue.cc:260] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } }
I20260923 14:45:54.821568 4932 sys_catalog.cc:455] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } } }
I20260923 14:45:54.821760 4932 sys_catalog.cc:458] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:45:54.822284 4926 sys_catalog.cc:564] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 14:45:54.827224 4943 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:45:54.827307 4943 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:45:54.829023 4858 master_runner.cc:428] Detected that this master d393dc58431a42f98fc87bdf91dff6c3 is joining an existing cluster
I20260923 14:45:54.829071 4858 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.2.253:43049
I20260923 14:45:54.838728 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.831105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34870 (local address 127.30.2.254:32951)
0923 14:45:54.831236 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:54.831242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:54.831542 (+ 300us) server_negotiation.cc:407] Connection header received
0923 14:45:54.831688 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:54.831692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:54.831749 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:54.831866 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:54.832406 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.832926 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.833791 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.834012 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.834121 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.835243 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:54.835247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:54.835250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:54.835279 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:54.836770 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.837343 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.837348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.837352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.837444 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.837764 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.837767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.837769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.838000 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:54.838078 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:54.838458 (+ 380us) server_negotiation.cc:299] Negotiation successful
0923 14:45:54.838573 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:45:54.839309 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.831039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:34870)
0923 14:45:54.831458 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:54.831471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:54.831589 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:54.831998 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:54.832003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:54.832182 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:54.832283 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.832288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.833030 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.833036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.833666 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.833674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.834131 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.834135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.834227 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.834762 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:54.834784 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:54.835099 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:54.836896 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.836900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.836902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.837094 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.837551 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.837555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.837556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.837657 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.838163 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:54.838167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:54.838273 (+ 106us) client_negotiation.cc:769] Sending connection context
0923 14:45:54.838932 (+ 659us) client_negotiation.cc:240] Negotiation successful
0923 14:45:54.839078 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":238,"threads_started":1}
I20260923 14:45:54.847082 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.837438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45666 (local address 127.30.2.253:43049)
0923 14:45:54.837555 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:54.837559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:54.837571 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:45:54.837603 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:54.837605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:54.837657 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:54.837730 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:54.838302 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.838996 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.839990 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.840298 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.840375 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.842442 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:54.842447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:54.842452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:54.842478 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:54.845217 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.845930 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.845936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.845939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.846018 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.846290 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.846294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.846295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.846453 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:54.846551 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:54.846823 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 14:45:54.846960 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":49}
I20260923 14:45:54.847095 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.831162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43049 (local address 127.0.0.1:45666)
0923 14:45:54.831818 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:54.831832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:54.831900 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:54.837854 (+ 5954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:54.837857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:54.838080 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:54.838197 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.838203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.839162 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.839166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.839869 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.839881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.840969 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.840974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.841077 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.841597 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:54.841610 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:54.842021 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:54.845328 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.845331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.845333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.845835 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.846111 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.846115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.846117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.846198 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.846550 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:54.846553 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:54.846638 (+ 85us) client_negotiation.cc:769] Sending connection context
0923 14:45:54.846755 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 14:45:54.846909 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":87,"threads_started":1}
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.254@KRBTEST.COM for oryx/127.30.2.253@KRBTEST.COM
I20260923 14:45:54.862090 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.845422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.30.2.253:43049)
0923 14:45:54.845743 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:54.845753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:54.852232 (+ 6479us) server_negotiation.cc:407] Connection header received
0923 14:45:54.852471 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:54.852475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:54.852560 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:54.852661 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:54.853575 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.854428 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.855804 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.856225 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.856367 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.858804 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:54.858810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:54.858813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:54.858835 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:54.860326 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.860793 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.860799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.860800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.860881 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.861180 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.861183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.861185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.861346 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:54.861450 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:54.861811 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 14:45:54.861925 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":137,"threads_started":1}
I20260923 14:45:54.862103 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.844685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43049 (local address 127.0.0.1:45676)
0923 14:45:54.852119 (+ 7434us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:54.852162 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:54.852297 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:54.852756 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:54.852763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:54.853247 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:54.853423 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.853434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.854542 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.854548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.855632 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.855657 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.856361 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.856364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.856440 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.857087 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:54.857099 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:54.858661 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:54.860436 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.860447 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.860454 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.860691 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.860970 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.860974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.860976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.861088 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.861450 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:54.861458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:54.861576 (+ 118us) client_negotiation.cc:769] Sending connection context
0923 14:45:54.861726 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 14:45:54.861950 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7206,"thread_start_us":98,"threads_started":1}
I20260923 14:45:54.863734 4394 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.2.253:43049: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3"
member_type: NON_VOTER
last_known_addr {
host: "127.30.2.253"
port: 43049
}
attrs {
promote: true
}
}
dest_uuid: "6ba2e576559541c8b9fe54558d267710"
cas_config_opid_index: -1
I20260923 14:45:54.864411 4394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.30.2.252@KRBTEST.COM; defaulting to no policy
I20260923 14:45:54.866016 4955 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d393dc58431a42f98fc87bdf91dff6c3 (127.30.2.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } } }
I20260923 14:45:54.866986 4900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:54.867872 4955 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.2.253:43049
I20260923 14:45:54.868271 4858 master.cc:537] Master@127.30.2.253:43049 shutting down...
I20260923 14:45:54.868319 4954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } } }
I20260923 14:45:54.868407 4954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:54.868628 4900 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 FOLLOWER]: Refusing update from remote peer 6ba2e576559541c8b9fe54558d267710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.252@KRBTEST.COM" created.
I20260923 14:45:54.869351 4954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 14:45:54.871506 4378 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 14:45:54.872387 4378 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 -> Peer d393dc58431a42f98fc87bdf91dff6c3 (127.30.2.253:43049): Couldn't send request to peer d393dc58431a42f98fc87bdf91dff6c3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 14:45:54.874090 4858 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:54.874379 4858 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:54.874490 4858 tablet_replica.cc:330] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: stopping tablet replica
I20260923 14:45:54.879424 4858 master.cc:552] Master@127.30.2.253:43049 shutdown complete.
I20260923 14:45:54.879499 4858 master_runner.cc:305] Clearing existing system tablet
I20260923 14:45:54.879738 4858 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:54.879842 4858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:54.880913 4858 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:54.881773 4960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.881980 4858 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:54.882118 4858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:54.882169 4858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/wal
uuid: "d393dc58431a42f98fc87bdf91dff6c3"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "7c1db8f3a2a968194ea43a24c82cb3b8"
server_key_iv: "aa6154b123137db424e7fda031f82c06"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.883873 4858 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:54.886777 4858 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 14:45:54.886888 4858 log.cc:1196] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 14:45:54.887676 4858 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: Deleting consensus metadata
I20260923 14:45:54.887888 4858 master_runner.cc:315] Copying system tablet from 127.30.2.254:32951
I20260923 14:45:54.889096 4858 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: tablet copy: Beginning tablet copy session from remote peer at address 127.30.2.254:32951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.252.keytab.
Entry for principal oryx/127.30.2.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.252.keytab.
I20260923 14:45:54.893247 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.252:38759
--webserver_interface=127.30.2.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.252.keytab
--principal=oryx/127.30.2.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:32951,127.30.2.253:43049,127.30.2.252:38759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:45:54.897991 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.889773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.30.2.254:32951)
0923 14:45:54.889924 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:54.889930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:54.889998 (+ 68us) server_negotiation.cc:407] Connection header received
0923 14:45:54.890172 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:54.890175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:54.890231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:54.890313 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:54.890778 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.891367 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.892644 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.892870 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.892927 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.894210 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:54.894214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:54.894217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:54.894253 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:54.896229 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.896747 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.896751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.896752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.896839 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:54.897157 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:54.897160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:54.897162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:54.897342 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:54.897450 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:54.897722 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 14:45:54.897832 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 14:45:54.898000 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:54.889665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:34884)
0923 14:45:54.889914 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:54.889928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:54.890045 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:54.890330 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:54.890333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:54.890518 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:54.890664 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.890670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.891482 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.891486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.892500 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:54.892515 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:54.893053 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:54.893055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:54.893133 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:54.893720 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:54.893734 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:54.894064 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:54.896379 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.896384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.896386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.896635 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.896972 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:54.896975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:54.896977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:54.897061 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:54.897438 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:54.897441 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:54.897551 (+ 110us) client_negotiation.cc:769] Sending connection context
0923 14:45:54.897668 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 14:45:54.897806 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":105,"threads_started":1}
I20260923 14:45:54.898782 4414 tablet_copy_service.cc:140] P 6ba2e576559541c8b9fe54558d267710: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d393dc58431a42f98fc87bdf91dff6c3 ({username='oryx', principal='oryx/127.30.2.253@KRBTEST.COM'} at 127.0.0.1:34884)
I20260923 14:45:54.898955 4414 tablet_copy_service.cc:161] P 6ba2e576559541c8b9fe54558d267710: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d393dc58431a42f98fc87bdf91dff6c3 at {username='oryx', principal='oryx/127.30.2.253@KRBTEST.COM'} at 127.0.0.1:34884: session id = d393dc58431a42f98fc87bdf91dff6c3-00000000000000000000000000000000
I20260923 14:45:54.900471 4414 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 14:45:54.901873 4858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 14:45:54.903656 4858 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: tablet copy: Starting download of 0 data blocks...
I20260923 14:45:54.903878 4858 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: tablet copy: Starting download of 1 WAL segments...
I20260923 14:45:54.905463 4858 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 14:45:54.906966 4414 tablet_copy_service.cc:342] P 6ba2e576559541c8b9fe54558d267710: Request end of tablet copy session d393dc58431a42f98fc87bdf91dff6c3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.2.253@KRBTEST.COM'} at 127.0.0.1:34884
I20260923 14:45:54.907081 4414 tablet_copy_service.cc:433] P 6ba2e576559541c8b9fe54558d267710: ending tablet copy session d393dc58431a42f98fc87bdf91dff6c3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d393dc58431a42f98fc87bdf91dff6c3
I20260923 14:45:54.914194 4858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:54.915763 4968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:54.915810 4969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:54.916064 4858 server_base.cc:1034] running on GCE node
W20260923 14:45:54.916543 4971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:54.916769 4858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:54.917061 4858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:54.918226 4858 hybrid_clock.cc:648] HybridClock initialized: now 1790174754918210 us; error 30 us; skew 500 ppm
Sep 23 14:45:54 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174754, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:54.919628 4858 init.cc:377] Logged in from keytab as oryx/127.30.2.253@KRBTEST.COM (short username oryx)
I20260923 14:45:54.920125 4858 webserver.cc:460] Webserver started at http://127.30.2.253:34211/ using document root <none> and password file <none>
I20260923 14:45:54.920342 4858 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:54.920384 4858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:54.921541 4858 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 14:45:54.922431 4978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:54.922663 4858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 14:45:54.922752 4858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/wal
uuid: "d393dc58431a42f98fc87bdf91dff6c3"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "7c1db8f3a2a968194ea43a24c82cb3b8"
server_key_iv: "aa6154b123137db424e7fda031f82c06"
server_key_version: "encryptionkey@0"
I20260923 14:45:54.922827 4858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:54.929003 4858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:54.930938 4858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:54.935614 4858 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.253:43049
I20260923 14:45:54.936187 4858 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
I20260923 14:45:54.937674 5031 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:45:54.938550 5031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: Bootstrap starting.
I20260923 14:45:54.940162 5030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.253:43049 every 8 connection(s)
I20260923 14:45:54.946722 5031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:45:54.947038 5031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: Bootstrap complete.
I20260923 14:45:54.947512 5031 raft_consensus.cc:348] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } }
I20260923 14:45:54.947652 5031 raft_consensus.cc:729] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d393dc58431a42f98fc87bdf91dff6c3, State: Initialized, Role: LEARNER
I20260923 14:45:54.947814 5031 consensus_queue.cc:260] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } }
I20260923 14:45:54.948151 5034 sys_catalog.cc:455] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } } }
I20260923 14:45:54.948319 5034 sys_catalog.cc:458] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: This master's current role is: LEARNER
I20260923 14:45:54.948324 5031 sys_catalog.cc:564] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:54.951157 5045 catalog_manager.cc:1259] Loaded cluster ID: ea6a320ed7bf45599adce879d795567e
I20260923 14:45:54.951247 5045 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: loading cluster ID for follower catalog manager: success
I20260923 14:45:54.953003 5045 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: acquiring CA information for follower catalog manager: success
I20260923 14:45:54.954078 5045 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 14:45:55.052003 4967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:55.052245 4967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:55.052309 4967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:55.056255 4967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:55.056299 4967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:55.056341 4967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:55.056371 4967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:55.056401 4967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:55.061697 4967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:32951,127.30.2.253:43049,127.30.2.252:38759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.252.keytab
--principal=oryx/127.30.2.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.252:38759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.2.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:55.062700 4967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:55.063712 4967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:55.072508 5053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:55.072515 5050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:55.072515 5051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:55.072963 4967 server_base.cc:1034] running on GCE node
I20260923 14:45:55.073609 4967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:55.074699 4967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:55.075882 4967 hybrid_clock.cc:648] HybridClock initialized: now 1790174755075859 us; error 34 us; skew 500 ppm
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:55.078676 4967 init.cc:377] Logged in from keytab as oryx/127.30.2.252@KRBTEST.COM (short username oryx)
I20260923 14:45:55.079469 4967 webserver.cc:460] Webserver started at http://127.30.2.252:34629/ using document root <none> and password file <none>
I20260923 14:45:55.080037 4967 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:55.080140 4967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:55.080374 4967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:55.082283 4967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data/instance:
uuid: "bceaed5b40c94555bf6695b45659ae09"
format_stamp: "Formatted at 2026-09-23 14:45:55 on dist-test-slave-wjf2"
server_key: "de2e505b2eb4ebf5c099c125035ed78f"
server_key_iv: "97891992f6908746ddb7a4b0e6f930f9"
server_key_version: "encryptionkey@0"
I20260923 14:45:55.082775 4967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/wal/instance:
uuid: "bceaed5b40c94555bf6695b45659ae09"
format_stamp: "Formatted at 2026-09-23 14:45:55 on dist-test-slave-wjf2"
server_key: "de2e505b2eb4ebf5c099c125035ed78f"
server_key_iv: "97891992f6908746ddb7a4b0e6f930f9"
server_key_version: "encryptionkey@0"
I20260923 14:45:55.086417 4967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260923 14:45:55.088757 5060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.089995 4967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:45:55.090090 4967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/wal
uuid: "bceaed5b40c94555bf6695b45659ae09"
format_stamp: "Formatted at 2026-09-23 14:45:55 on dist-test-slave-wjf2"
server_key: "de2e505b2eb4ebf5c099c125035ed78f"
server_key_iv: "97891992f6908746ddb7a4b0e6f930f9"
server_key_version: "encryptionkey@0"
I20260923 14:45:55.090193 4967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:55.115299 4967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:55.118434 4967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:55.118579 4967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:55.125020 4585 heartbeater.cc:503] Master 127.30.2.254:32951 was elected leader, sending a full tablet report...
I20260923 14:45:55.126377 4967 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.252:38759
I20260923 14:45:55.126725 5112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.252:38759 every 8 connection(s)
I20260923 14:45:55.127406 4967 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
I20260923 14:45:55.130228 5113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:55.131006 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 4967
I20260923 14:45:55.134975 5113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.2.252@KRBTEST.COM
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.252@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:45:55.145924 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.132486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.30.2.252:38759)
0923 14:45:55.132907 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.132921 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.132955 (+ 34us) server_negotiation.cc:407] Connection header received
0923 14:45:55.133579 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.133596 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.133914 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.134246 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.135046 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.136149 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.137271 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.137969 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.138146 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.140782 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.140806 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.140819 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.140865 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.142713 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.143208 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.143215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.143222 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.143360 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.143667 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.143669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.143671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.144127 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.144320 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.144673 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.144937 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"spinlock_wait_cycles":2176,"thread_start_us":128,"threads_started":1}
I20260923 14:45:55.146116 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.136576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:34898)
0923 14:45:55.136964 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.137015 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.137128 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.137435 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.137441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.137844 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.137990 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.137998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.139014 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.139017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.139672 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.139681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.140160 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.140165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.140294 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.140843 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.140859 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.142623 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.144564 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.144570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.144575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.144841 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.145104 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.145108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.145109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.145202 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.145522 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.145527 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.145651 (+ 124us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.145790 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.145953 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":141,"threads_started":1}
I20260923 14:45:55.147003 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.136783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.30.2.254:32951)
0923 14:45:55.136909 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.136913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.137096 (+ 183us) server_negotiation.cc:407] Connection header received
0923 14:45:55.137227 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.137230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.137289 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.137358 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.138129 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.138890 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.139815 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.140020 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.140127 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.142765 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.142771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.142775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.142802 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.144465 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.144938 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.144941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.144943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.145016 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.145284 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.145286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.145287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.145431 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.145499 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.145840 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.145924 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 14:45:55.147933 5113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.252@KRBTEST.COM for oryx/127.30.2.253@KRBTEST.COM
I20260923 14:45:55.155740 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.148355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43049 (local address 127.0.0.1:45678)
0923 14:45:55.148509 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.148524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.148593 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.148998 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.149001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.149136 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.149229 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.149234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.150314 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.150317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.150801 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.150808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.151229 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.151232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.151297 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.151687 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.151697 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.152845 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.154602 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.154605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.154606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.154788 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.155012 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.155014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.155016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.155085 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.155391 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.155393 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.155460 (+ 67us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.155538 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.155638 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 14:45:55.156002 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.148589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.30.2.253:43049)
0923 14:45:55.148810 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.148813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.148824 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:45:55.148857 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.148859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.148894 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.148963 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.149338 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.150210 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.150897 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.151132 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.151222 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.152982 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.152985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.152987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.153009 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.154512 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.154868 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.154870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.154871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.154939 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.155159 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.155161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.155162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.155314 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.155386 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.155652 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.155753 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20260923 14:45:55.156759 5113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.252@KRBTEST.COM for oryx/127.30.2.252@KRBTEST.COM
I20260923 14:45:55.164536 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.157287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:38759 (local address 127.0.0.1:34792)
0923 14:45:55.157415 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.157426 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.157481 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.157721 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.157727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.157853 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.157916 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.157920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.158961 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.158964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.159464 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.159471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.159975 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.159978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.160032 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.160411 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.160421 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.161495 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.163174 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.163179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.163181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.163359 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.163613 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.163618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.163620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.163702 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.164060 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.164063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.164139 (+ 76us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.164241 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.164368 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 14:45:55.164548 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.157336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34792 (local address 127.30.2.252:38759)
0923 14:45:55.157454 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.157457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.157467 (+ 10us) server_negotiation.cc:407] Connection header received
0923 14:45:55.157533 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.157534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.157566 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.157624 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.158017 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.158861 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.159580 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.159862 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.159950 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.161617 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.161620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.161623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.161650 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.163026 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.163449 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.163454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.163456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.163526 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.163811 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.163813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.163815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.163966 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.164069 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.164295 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.164420 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":58}
I20260923 14:45:55.167398 5113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: Bootstrap starting.
I20260923 14:45:55.169710 5113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:55.170363 5113 log.cc:824] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:55.172272 5113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: No bootstrap required, opened a new log
I20260923 14:45:55.175089 5113 raft_consensus.cc:348] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } }
I20260923 14:45:55.175271 5113 raft_consensus.cc:374] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:55.175308 5113 raft_consensus.cc:729] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceaed5b40c94555bf6695b45659ae09, State: Initialized, Role: FOLLOWER
I20260923 14:45:55.175650 5113 consensus_queue.cc:260] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } }
I20260923 14:45:55.176082 5119 sys_catalog.cc:455] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } } }
I20260923 14:45:55.176234 5119 sys_catalog.cc:458] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:45:55.176764 5113 sys_catalog.cc:564] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 14:45:55.181736 5130 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:45:55.181865 5130 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:45:55.185477 4967 master_runner.cc:428] Detected that this master bceaed5b40c94555bf6695b45659ae09 is joining an existing cluster
I20260923 14:45:55.185554 4967 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.2.252:38759
I20260923 14:45:55.197440 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.188264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.30.2.252:38759)
0923 14:45:55.188365 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.188370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.188454 (+ 84us) server_negotiation.cc:407] Connection header received
0923 14:45:55.188611 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.188614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.188649 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.188703 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.189671 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.190471 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.191893 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.192217 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.192288 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.193841 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.193845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.193848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.193872 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.195550 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.196149 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.196154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.196156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.196240 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.196552 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.196555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.196557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.196733 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.196846 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.197130 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.197266 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":43}
I20260923 14:45:55.197456 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.187577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:38759 (local address 127.0.0.1:34806)
0923 14:45:55.188379 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.188394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.188515 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.188839 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.188842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.189239 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.189531 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.189543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.190628 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.190632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.191720 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.191732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.192445 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.192449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.192576 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.193212 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.193224 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.193699 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.195686 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.195691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.195694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.196017 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.196344 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.196350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.196352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.196445 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.196834 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.196837 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.196935 (+ 98us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.197055 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.197211 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":142,"spinlock_wait_cycles":5376,"thread_start_us":454,"threads_started":1}
I20260923 14:45:55.199101 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 4456
I20260923 14:45:55.206295 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:42417
--local_ip_for_outbound_sockets=127.30.2.193
--tserver_master_addrs=127.30.2.254:32951
--webserver_port=41143
--webserver_interface=127.30.2.193
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab
--principal=oryx/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.2.254:32951,127.30.2.253:43049
--tserver_master_addrs=127.30.2.254:32951,127.30.2.253:43049,127.30.2.252:38759 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:45:55.213629 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.187649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.30.2.253:43049)
0923 14:45:55.187820 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.187828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.188373 (+ 545us) server_negotiation.cc:407] Connection header received
0923 14:45:55.188594 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.188599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.188651 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.188705 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.189721 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.190492 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.192104 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.192458 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.192555 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.209239 (+ 16684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.209247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.209252 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.209294 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.211262 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.211878 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.211883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.211885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.211991 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.212401 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.212406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.212409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.212711 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.212818 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.213234 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.213392 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:45:55.214268 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.187569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43049 (local address 127.0.0.1:45690)
0923 14:45:55.188285 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.188302 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.188454 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.188839 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.188842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.189239 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.189581 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.189593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.190633 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.190635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.191973 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.191983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.192724 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.192730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.192862 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.208350 (+ 15488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.208369 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.209000 (+ 631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.211397 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.211403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.211408 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.211704 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.212134 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.212139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.212143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.212248 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.212890 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.212893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.213018 (+ 125us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.213866 (+ 848us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.214032 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":161,"thread_start_us":436,"threads_started":1}
I20260923 14:45:55.215476 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.188065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:34912)
0923 14:45:55.188481 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.188494 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.188638 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.188977 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.188980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.189250 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.189567 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.189585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.190666 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.190669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.191663 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.191674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.192444 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.192447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.192603 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.210537 (+ 17934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.210559 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.211011 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.213373 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.213376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.213378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.213616 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.214250 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.214253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.214256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.214384 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.214907 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.214910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.215009 (+ 99us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.215119 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.215273 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":188,"spinlock_wait_cycles":1280,"thread_start_us":225,"threads_started":1}
I20260923 14:45:55.216091 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.187877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.30.2.254:32951)
0923 14:45:55.188024 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.188030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.188562 (+ 532us) server_negotiation.cc:407] Connection header received
0923 14:45:55.188745 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.188748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.188793 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.188842 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.189714 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.190514 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.191859 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.192193 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.192272 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.211245 (+ 18973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.211250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.211253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.211288 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.213242 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.214002 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.214006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.214008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.214124 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.214482 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.214487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.214489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.214750 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.214855 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.215727 (+ 872us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.215923 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174753, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.254@KRBTEST.COM for oryx/127.30.2.252@KRBTEST.COM
I20260923 14:45:55.228667 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.219905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:38759 (local address 127.0.0.1:34812)
0923 14:45:55.220046 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.220061 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.220154 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.220432 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.220436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.220627 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.220716 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.220722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.221662 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.221666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.222255 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.222264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.222798 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.222809 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.222913 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.223453 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.223472 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.225065 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.226931 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.226935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.226937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.227153 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.227484 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.227488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.227490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.227593 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.228093 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.228096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.228207 (+ 111us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.228324 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.228479 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 14:45:55.228665 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.219937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.30.2.252:38759)
0923 14:45:55.220053 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.220058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.220111 (+ 53us) server_negotiation.cc:407] Connection header received
0923 14:45:55.220268 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.220271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.220317 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.220414 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.220841 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.221540 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.222392 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.222664 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.222783 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.225217 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.225222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.225225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.225253 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.226780 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.227261 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.227266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.227268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.227356 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.227705 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.227709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.227711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.227960 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.228100 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.228394 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.228506 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:45:55.230018 4394 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.2.252:38759: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bceaed5b40c94555bf6695b45659ae09"
member_type: NON_VOTER
last_known_addr {
host: "127.30.2.252"
port: 38759
}
attrs {
promote: true
}
}
dest_uuid: "6ba2e576559541c8b9fe54558d267710"
cas_config_opid_index: 5
I20260923 14:45:55.230669 4394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } }
I20260923 14:45:55.231959 4955 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bceaed5b40c94555bf6695b45659ae09 (127.30.2.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.233300 4955 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.2.252:38759
I20260923 14:45:55.233491 4955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.233624 4955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:55.234100 4967 master.cc:537] Master@127.30.2.252:38759 shutting down...
I20260923 14:45:55.234591 5087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:55.235826 5087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 FOLLOWER]: Refusing update from remote peer 6ba2e576559541c8b9fe54558d267710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 14:45:55.236507 4954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:55.242790 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.232106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43049 (local address 127.0.0.1:45706)
0923 14:45:55.237006 (+ 4900us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.237024 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.237177 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.237693 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.237698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.237723 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:55.237826 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.237834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.239277 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.239282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.240749 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.240765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.242244 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.242250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.242382 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.242448 (+ 66us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.242534 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.242628 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4826}
I20260923 14:45:55.243275 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.232373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.30.2.253:43049)
0923 14:45:55.236906 (+ 4533us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.236912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.237101 (+ 189us) server_negotiation.cc:407] Connection header received
0923 14:45:55.237315 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.237319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.237397 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.237480 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:55.238219 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.239095 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.240927 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.241818 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.241920 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.242120 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.243056 (+ 936us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.243137 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4430}
I20260923 14:45:55.244798 5005 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 LEARNER]: Refusing update from remote peer 6ba2e576559541c8b9fe54558d267710: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 14:45:55.245513 4954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 14:45:55.247918 5005 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bceaed5b40c94555bf6695b45659ae09 (127.30.2.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.250514 5034 sys_catalog.cc:455] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba2e576559541c8b9fe54558d267710. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.250643 5034 sys_catalog.cc:458] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: This master's current role is: LEARNER
I20260923 14:45:55.250759 5034 sys_catalog.cc:455] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: NON_VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: true } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.250842 5034 sys_catalog.cc:458] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: This master's current role is: LEARNER
W20260923 14:45:55.251993 4380 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 -> Peer bceaed5b40c94555bf6695b45659ae09 (127.30.2.252:38759): Couldn't send request to peer bceaed5b40c94555bf6695b45659ae09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 14:45:55.252084 4954 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710: attempting to promote NON_VOTER d393dc58431a42f98fc87bdf91dff6c3 to VOTER
I20260923 14:45:55.252499 4954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } }
I20260923 14:45:55.254043 5004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 LEARNER]: Refusing update from remote peer 6ba2e576559541c8b9fe54558d267710: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260923 14:45:55.254621 4380 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 -> Peer bceaed5b40c94555bf6695b45659ae09 (127.30.2.252:38759): Couldn't send request to peer bceaed5b40c94555bf6695b45659ae09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 14:45:55.254913 4955 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 14:45:55.257380 4954 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d393dc58431a42f98fc87bdf91dff6c3 (127.30.2.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.257912 5004 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d393dc58431a42f98fc87bdf91dff6c3 (127.30.2.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.259429 4954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.259608 4954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:55.259532 5034 sys_catalog.cc:455] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.259656 5034 sys_catalog.cc:458] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:45:55.259972 4954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.260159 4954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:55.261391 4967 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:55.261762 4967 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:55.261901 4967 tablet_replica.cc:330] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: stopping tablet replica
I20260923 14:45:55.266465 4967 master.cc:552] Master@127.30.2.252:38759 shutdown complete.
I20260923 14:45:55.266539 4967 master_runner.cc:305] Clearing existing system tablet
I20260923 14:45:55.266808 4967 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:55.266882 4967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:55.268095 4967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 14:45:55.268945 5144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.269276 4967 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:55.269436 4967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:55.269479 4967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/wal
uuid: "bceaed5b40c94555bf6695b45659ae09"
format_stamp: "Formatted at 2026-09-23 14:45:55 on dist-test-slave-wjf2"
server_key: "de2e505b2eb4ebf5c099c125035ed78f"
server_key_iv: "97891992f6908746ddb7a4b0e6f930f9"
server_key_version: "encryptionkey@0"
I20260923 14:45:55.271278 4967 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:55.273406 4967 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 14:45:55.273512 4967 log.cc:1196] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 14:45:55.273772 4967 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: Deleting consensus metadata
I20260923 14:45:55.273981 4967 master_runner.cc:315] Copying system tablet from 127.30.2.254:32951
I20260923 14:45:55.275147 4967 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: tablet copy: Beginning tablet copy session from remote peer at address 127.30.2.254:32951
I20260923 14:45:55.284175 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.275773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.30.2.254:32951)
0923 14:45:55.275966 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.275974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.276206 (+ 232us) server_negotiation.cc:407] Connection header received
0923 14:45:55.276373 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.276376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.276426 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.276528 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.277103 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.277771 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.278838 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.279090 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.279213 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.280616 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.280621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.280624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.280651 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.282295 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.282769 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.282774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.282776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.282866 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.283205 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.283208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.283210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.283401 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.283515 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.283872 (+ 357us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.283996 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 14:45:55.284348 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.275724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:34918)
0923 14:45:55.276105 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.276120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.276256 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.276567 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.276571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.276812 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.276985 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.276997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.277901 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.277907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.278718 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.278733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.279213 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.279218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.279320 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.280033 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.280055 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.280456 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.282415 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.282422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.282425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.282662 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.282970 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.282973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.282975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.283074 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.283515 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.283519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.283632 (+ 113us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.283746 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.283919 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":126,"threads_started":1}
I20260923 14:45:55.284987 4414 tablet_copy_service.cc:140] P 6ba2e576559541c8b9fe54558d267710: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bceaed5b40c94555bf6695b45659ae09 ({username='oryx', principal='oryx/127.30.2.252@KRBTEST.COM'} at 127.0.0.1:34918)
I20260923 14:45:55.285133 4414 tablet_copy_service.cc:161] P 6ba2e576559541c8b9fe54558d267710: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bceaed5b40c94555bf6695b45659ae09 at {username='oryx', principal='oryx/127.30.2.252@KRBTEST.COM'} at 127.0.0.1:34918: session id = bceaed5b40c94555bf6695b45659ae09-00000000000000000000000000000000
I20260923 14:45:55.286158 4414 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 14:45:55.287420 4967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 14:45:55.289384 4967 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: tablet copy: Starting download of 0 data blocks...
I20260923 14:45:55.289597 4967 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: tablet copy: Starting download of 1 WAL segments...
I20260923 14:45:55.291052 4967 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 14:45:55.292703 4414 tablet_copy_service.cc:342] P 6ba2e576559541c8b9fe54558d267710: Request end of tablet copy session bceaed5b40c94555bf6695b45659ae09-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.2.252@KRBTEST.COM'} at 127.0.0.1:34918
I20260923 14:45:55.292804 4414 tablet_copy_service.cc:433] P 6ba2e576559541c8b9fe54558d267710: ending tablet copy session bceaed5b40c94555bf6695b45659ae09-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bceaed5b40c94555bf6695b45659ae09
I20260923 14:45:55.299608 4967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:55.301467 5154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:55.301605 5151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:55.301647 5152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:55.302130 4967 server_base.cc:1034] running on GCE node
I20260923 14:45:55.302305 4967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:55.302657 4967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:55.303877 4967 hybrid_clock.cc:648] HybridClock initialized: now 1790174755303884 us; error 44 us; skew 500 ppm
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:55.305481 4967 init.cc:377] Logged in from keytab as oryx/127.30.2.252@KRBTEST.COM (short username oryx)
I20260923 14:45:55.305972 4967 webserver.cc:460] Webserver started at http://127.30.2.252:35433/ using document root <none> and password file <none>
I20260923 14:45:55.306188 4967 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:55.306247 4967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:55.307480 4967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:55.308578 5161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.308818 4967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 14:45:55.308967 4967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/wal
uuid: "bceaed5b40c94555bf6695b45659ae09"
format_stamp: "Formatted at 2026-09-23 14:45:55 on dist-test-slave-wjf2"
server_key: "de2e505b2eb4ebf5c099c125035ed78f"
server_key_iv: "97891992f6908746ddb7a4b0e6f930f9"
server_key_version: "encryptionkey@0"
I20260923 14:45:55.309127 4967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:55.322356 4967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:55.324640 4967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:55.330229 4967 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.252:38759
I20260923 14:45:55.330253 5213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.252:38759 every 8 connection(s)
I20260923 14:45:55.330855 4967 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
I20260923 14:45:55.332647 5214 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:45:55.333665 5214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: Bootstrap starting.
I20260923 14:45:55.344528 5214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:45:55.344832 5214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: Bootstrap complete.
I20260923 14:45:55.345359 5214 raft_consensus.cc:348] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } }
I20260923 14:45:55.345500 5214 raft_consensus.cc:729] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceaed5b40c94555bf6695b45659ae09, State: Initialized, Role: LEARNER
I20260923 14:45:55.345631 5214 consensus_queue.cc:260] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } }
I20260923 14:45:55.346004 5217 sys_catalog.cc:455] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.346182 5217 sys_catalog.cc:458] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: This master's current role is: LEARNER
I20260923 14:45:55.346197 5214 sys_catalog.cc:564] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:55.349853 5228 catalog_manager.cc:1259] Loaded cluster ID: ea6a320ed7bf45599adce879d795567e
I20260923 14:45:55.349949 5228 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: loading cluster ID for follower catalog manager: success
I20260923 14:45:55.351653 5228 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: acquiring CA information for follower catalog manager: success
I20260923 14:45:55.352941 5228 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:45:55.355310 4719 heartbeater.cc:503] Master 127.30.2.254:32951 was elected leader, sending a full tablet report...
W20260923 14:45:55.361613 5139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:55.361902 5139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:55.361960 5139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:55.367830 5139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:55.367892 5139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:55.367976 5139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:55.374385 5139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab
--principal=oryx/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:42417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=41143
--tserver_master_addrs=127.30.2.254:32951,127.30.2.253:43049,127.30.2.252:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:55.375442 5139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:55.376502 5139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:55.385893 5236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:55.385978 5234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:55.386188 5233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:55.386245 5139 server_base.cc:1034] running on GCE node
I20260923 14:45:55.386658 5139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:55.387305 5139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:55.388495 5139 hybrid_clock.cc:648] HybridClock initialized: now 1790174755388464 us; error 39 us; skew 500 ppm
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:55.391366 5139 init.cc:377] Logged in from keytab as oryx/127.30.2.193@KRBTEST.COM (short username oryx)
I20260923 14:45:55.392294 5139 webserver.cc:460] Webserver started at http://127.30.2.193:41143/ using document root <none> and password file <none>
I20260923 14:45:55.392867 5139 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:55.392974 5139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:55.397339 5139 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:45:55.399461 5243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.400729 5139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:55.400880 5139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "6c974d96d963445caf911d5aee86d92a"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "cbd94cb4b3d42f6b14cbfbb5815efc44"
server_key_iv: "e0d8072ae4b65adc2cb79868e0c13f4c"
server_key_version: "encryptionkey@0"
I20260923 14:45:55.401412 5139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:55.418001 5139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:55.421581 5139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:55.421803 5139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:55.422472 5139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:55.423418 5139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:55.423501 5139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.423571 5139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:55.423610 5139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.433418 5139 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:42417
I20260923 14:45:55.433434 5356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:42417 every 8 connection(s)
I20260923 14:45:55.434530 5139 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:45:55.444376 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 5139
I20260923 14:45:55.444614 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 4590
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.252@KRBTEST.COM
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.253@KRBTEST.COM
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:45:55.453127 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.437242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:41837 (local address 127.30.2.252:38759)
0923 14:45:55.437601 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.437606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.438084 (+ 478us) server_negotiation.cc:407] Connection header received
0923 14:45:55.439158 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.439163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.439229 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.439357 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.440745 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.441760 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.442909 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.443155 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.443284 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.447522 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.447530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.447536 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.447573 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.450316 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.451163 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.451170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.451173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.451272 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.451645 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.451649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.451651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.451954 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.452035 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.452787 (+ 752us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.452937 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":68,"threads_started":1}
I20260923 14:45:55.453126 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.437052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:52865 (local address 127.30.2.254:32951)
0923 14:45:55.437456 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.437460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.438035 (+ 575us) server_negotiation.cc:407] Connection header received
0923 14:45:55.439290 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.439292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.439339 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.439434 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.440697 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.441559 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.443774 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.444111 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.444229 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.448113 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.448116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.448119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.448148 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.451117 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.451645 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.451648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.451650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.451758 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.452134 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.452138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.452140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.452367 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.452450 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.452827 (+ 377us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.452937 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317}
I20260923 14:45:55.453171 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.436569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:55421 (local address 127.30.2.253:43049)
0923 14:45:55.436714 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.436719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.438201 (+ 1482us) server_negotiation.cc:407] Connection header received
0923 14:45:55.439559 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.439563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.439618 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.439709 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.440740 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.441348 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.443240 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.443541 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.443646 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.448060 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.448066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.448073 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.448101 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.450439 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.451115 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.451121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.451126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.451213 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.451643 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.451650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.451654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.451953 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.452032 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.452785 (+ 753us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.452932 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 14:45:55.454952 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:41339
--local_ip_for_outbound_sockets=127.30.2.194
--tserver_master_addrs=127.30.2.254:32951
--webserver_port=36453
--webserver_interface=127.30.2.194
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab
--principal=oryx/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.2.254:32951,127.30.2.253:43049
--tserver_master_addrs=127.30.2.254:32951,127.30.2.253:43049,127.30.2.252:38759 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:45:55.455041 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.436882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43049 (local address 127.30.2.193:55421)
0923 14:45:55.438086 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.438102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.438902 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.439748 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.439751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.440153 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.440571 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.440581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.441553 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.441556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.443108 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.443121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.443655 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.443657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.443736 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.444580 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.444608 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.447751 (+ 3143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.450578 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.450584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.450589 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.450999 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.451325 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.451330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.451334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.451530 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.452117 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.452126 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.452487 (+ 361us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.453384 (+ 897us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.453966 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"mutex_wait_us":553,"spinlock_wait_cycles":85760,"thread_start_us":119,"threads_started":1}
I20260923 14:45:55.455354 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.436881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.193:52865)
0923 14:45:55.437855 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.437910 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.438885 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.439538 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.439547 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.440065 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.440551 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.440565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.441670 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.441673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.443108 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.443121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.444258 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.444262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.444360 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.446271 (+ 1911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.446286 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.447924 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.451243 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.451247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.451249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.451502 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.451912 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.451916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.451918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.452015 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.452533 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.452536 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.452631 (+ 95us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.453384 (+ 753us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.453869 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":270,"spinlock_wait_cycles":21120,"thread_start_us":704,"threads_started":1}
I20260923 14:45:55.455736 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.436882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:38759 (local address 127.30.2.193:41837)
0923 14:45:55.438003 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.438016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.438885 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.439558 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.439561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.440157 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.440571 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.440581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.441988 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.441992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.442781 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.442790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.443449 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.443453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.443653 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.444602 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.444616 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.447312 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.450495 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.450505 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.450522 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.451011 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.451405 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.451409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.451411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.451529 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.452128 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.452131 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.452492 (+ 361us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.454303 (+ 1811us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.454461 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":242,"spinlock_wait_cycles":111104,"thread_start_us":169,"threads_started":1}
I20260923 14:45:55.456408 5359 heartbeater.cc:347] Connected to a master server at 127.30.2.253:43049
I20260923 14:45:55.456614 5358 heartbeater.cc:347] Connected to a master server at 127.30.2.254:32951
I20260923 14:45:55.456673 5359 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:55.456736 5358 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:55.457253 5359 heartbeater.cc:511] Master 127.30.2.253:43049 requested a full tablet report, sending...
I20260923 14:45:55.457288 5358 heartbeater.cc:511] Master 127.30.2.254:32951 requested a full tablet report, sending...
I20260923 14:45:55.458055 5357 heartbeater.cc:347] Connected to a master server at 127.30.2.252:38759
I20260923 14:45:55.458137 5357 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:55.458293 5357 heartbeater.cc:511] Master 127.30.2.252:38759 requested a full tablet report, sending...
I20260923 14:45:55.458950 4991 ts_manager.cc:194] Registered new tserver with Master: 6c974d96d963445caf911d5aee86d92a (127.30.2.193:42417)
I20260923 14:45:55.459451 5178 ts_manager.cc:194] Registered new tserver with Master: 6c974d96d963445caf911d5aee86d92a (127.30.2.193:42417)
I20260923 14:45:55.459843 4394 ts_manager.cc:194] Re-registered known tserver with Master: 6c974d96d963445caf911d5aee86d92a (127.30.2.193:42417)
I20260923 14:45:55.460582 4394 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:52865
I20260923 14:45:55.566599 4853 heartbeater.cc:503] Master 127.30.2.254:32951 was elected leader, sending a full tablet report...
W20260923 14:45:55.569952 5365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:55.570170 5365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:55.570211 5365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:55.574071 5365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:55.574117 5365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:55.574172 5365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:55.585141 5365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab
--principal=oryx/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:41339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=36453
--tserver_master_addrs=127.30.2.254:32951,127.30.2.253:43049,127.30.2.252:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:55.586447 5365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:55.587831 5365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:55.597394 5373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:55.597399 5371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:55.597399 5370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:55.598199 5365 server_base.cc:1034] running on GCE node
I20260923 14:45:55.598626 5365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:55.599298 5365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:55.600481 5365 hybrid_clock.cc:648] HybridClock initialized: now 1790174755600468 us; error 38 us; skew 500 ppm
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:55.603168 5365 init.cc:377] Logged in from keytab as oryx/127.30.2.194@KRBTEST.COM (short username oryx)
I20260923 14:45:55.604107 5365 webserver.cc:460] Webserver started at http://127.30.2.194:36453/ using document root <none> and password file <none>
I20260923 14:45:55.604769 5365 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:55.604880 5365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:55.608843 5365 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260923 14:45:55.610904 5380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.612280 5365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:55.612409 5365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "1f02345dda9e49928bda61615714a6b8"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "456a46d39f9e444c7b1ab7e4e23c95a0"
server_key_iv: "d1471d63b7e9fdc48a93849cc718edc0"
server_key_version: "encryptionkey@0"
I20260923 14:45:55.612929 5365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:55.630544 5365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:55.633589 5365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:55.633793 5365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:55.634397 5365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:55.635263 5365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:55.635334 5365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.635407 5365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:55.635450 5365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.645511 5365 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:41339
I20260923 14:45:55.645570 5493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:41339 every 8 connection(s)
I20260923 14:45:55.646600 5365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:55.653477 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 5365
I20260923 14:45:55.653664 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 4724
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.253@KRBTEST.COM
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.252@KRBTEST.COM
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:45:55.664135 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:33897
--local_ip_for_outbound_sockets=127.30.2.195
--tserver_master_addrs=127.30.2.254:32951
--webserver_port=39433
--webserver_interface=127.30.2.195
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab
--principal=oryx/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.2.254:32951,127.30.2.253:43049
--tserver_master_addrs=127.30.2.254:32951,127.30.2.253:43049,127.30.2.252:38759 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:45:55.664458 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.648677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:44405 (local address 127.30.2.254:32951)
0923 14:45:55.649048 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.649053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.650262 (+ 1209us) server_negotiation.cc:407] Connection header received
0923 14:45:55.650894 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.650897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.650969 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.651085 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.652515 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.653067 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.654925 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.655349 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.655442 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.660366 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.660369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.660375 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.660409 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.662328 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.662853 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.662856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.662859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.662947 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.663262 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.663269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.663273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.663514 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.663585 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.664143 (+ 558us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.664271 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260923 14:45:55.664518 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.648675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:40707 (local address 127.30.2.252:38759)
0923 14:45:55.648940 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.648945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.650537 (+ 1592us) server_negotiation.cc:407] Connection header received
0923 14:45:55.650931 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.650935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.650992 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.651048 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.652528 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.653397 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.655023 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.655370 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.655438 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.659626 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.659631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.659634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.659663 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.661798 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.662443 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.662450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.662454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.662556 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.663032 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.663039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.663043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.663305 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.663381 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.664146 (+ 765us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.664287 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260923 14:45:55.664659 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.648910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:59641 (local address 127.30.2.253:43049)
0923 14:45:55.649036 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.649040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.649895 (+ 855us) server_negotiation.cc:407] Connection header received
0923 14:45:55.650931 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.650935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.650991 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.651050 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.652528 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.653399 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.654940 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.655287 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.655364 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.660767 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.660771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.660773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.660804 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.662838 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.663298 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.663303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.663307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.663380 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.663694 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.663698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.663699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.663950 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.664031 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.664398 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.664510 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:45:55.666275 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.649165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:38759 (local address 127.30.2.194:40707)
0923 14:45:55.650451 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.650469 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.650697 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.651261 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.651270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.651748 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.652370 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.652376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.653552 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.653555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.654907 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.654917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.655579 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.655582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.655966 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.656958 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.656986 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.659424 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.661937 (+ 2513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.661946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.661966 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.662328 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.662684 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.662689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.662694 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.662902 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.663465 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.663474 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.663849 (+ 375us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.665431 (+ 1582us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.665624 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1234,"mutex_wait_us":738,"spinlock_wait_cycles":38912,"thread_start_us":673,"threads_started":1}
I20260923 14:45:55.666301 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.649062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.194:44405)
0923 14:45:55.650168 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.650179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.650693 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.651260 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.651268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.651705 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.652356 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.652371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.653226 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.653231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.654775 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.654789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.655639 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.655646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.656018 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.657126 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.657147 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.660191 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.662463 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.662468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.662470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.662736 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.663061 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.663063 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.663065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.663149 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.663669 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.663673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.663854 (+ 181us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.664727 (+ 873us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.665226 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":284,"spinlock_wait_cycles":87424,"thread_start_us":154,"threads_started":1}
I20260923 14:45:55.667130 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.649062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43049 (local address 127.30.2.194:59641)
0923 14:45:55.649718 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.649786 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.650697 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.651261 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.651270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.651748 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.652367 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.652376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.653597 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.653599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.654796 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.654808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.655578 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.655582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.655969 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.656958 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.656986 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.659809 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.662950 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.662953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.662955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.663192 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.663494 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.663496 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.663499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.663580 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.664105 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.664108 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.664196 (+ 88us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.664833 (+ 637us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.665274 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":655,"spinlock_wait_cycles":38656,"thread_start_us":169,"threads_started":1}
I20260923 14:45:55.667693 5495 heartbeater.cc:347] Connected to a master server at 127.30.2.254:32951
I20260923 14:45:55.667754 5496 heartbeater.cc:347] Connected to a master server at 127.30.2.253:43049
I20260923 14:45:55.667714 5494 heartbeater.cc:347] Connected to a master server at 127.30.2.252:38759
I20260923 14:45:55.667970 5496 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:55.667971 5495 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:55.667970 5494 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:55.668607 5495 heartbeater.cc:511] Master 127.30.2.254:32951 requested a full tablet report, sending...
I20260923 14:45:55.668608 5494 heartbeater.cc:511] Master 127.30.2.252:38759 requested a full tablet report, sending...
I20260923 14:45:55.668608 5496 heartbeater.cc:511] Master 127.30.2.253:43049 requested a full tablet report, sending...
I20260923 14:45:55.669888 4991 ts_manager.cc:194] Registered new tserver with Master: 1f02345dda9e49928bda61615714a6b8 (127.30.2.194:41339)
I20260923 14:45:55.669864 5178 ts_manager.cc:194] Registered new tserver with Master: 1f02345dda9e49928bda61615714a6b8 (127.30.2.194:41339)
I20260923 14:45:55.669920 4394 ts_manager.cc:194] Re-registered known tserver with Master: 1f02345dda9e49928bda61615714a6b8 (127.30.2.194:41339)
I20260923 14:45:55.670701 4394 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:44405
W20260923 14:45:55.781476 5501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:55.781764 5501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:55.781854 5501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:55.785872 5501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:55.785950 5501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:55.786046 5501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:45:55.791339 5501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab
--principal=oryx/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:33897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=39433
--tserver_master_addrs=127.30.2.254:32951,127.30.2.253:43049,127.30.2.252:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:55.792512 5501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:55.793550 5501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:55.803318 5507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:55.803313 5506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:55.803326 5509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:55.803512 5501 server_base.cc:1034] running on GCE node
I20260923 14:45:55.804032 5501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:55.804769 5501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:55.805960 5501 hybrid_clock.cc:648] HybridClock initialized: now 1790174755805932 us; error 52 us; skew 500 ppm
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:55.808809 5501 init.cc:377] Logged in from keytab as oryx/127.30.2.195@KRBTEST.COM (short username oryx)
I20260923 14:45:55.809840 5501 webserver.cc:460] Webserver started at http://127.30.2.195:39433/ using document root <none> and password file <none>
I20260923 14:45:55.810457 5501 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:55.810560 5501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:55.814494 5501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:45:55.816710 5516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.818033 5501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:45:55.818224 5501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "5d3f26ce1e3a472281fbd8391587db1f"
format_stamp: "Formatted at 2026-09-23 14:45:54 on dist-test-slave-wjf2"
server_key: "6c26efd529fa8cf2858e5c7a549d7ffd"
server_key_iv: "0494184373a2bf55df13667c20e3b88f"
server_key_version: "encryptionkey@0"
I20260923 14:45:55.818775 5501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:55.832397 5501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:55.835764 5501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:55.836038 5501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:55.836690 5501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:45:55.837546 5501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:55.837615 5501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.837682 5501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:55.837734 5501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:55.847764 5501 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:33897
I20260923 14:45:55.847829 5630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:33897 every 8 connection(s)
I20260923 14:45:55.848839 5501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:45:55.854163 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 5501
I20260923 14:45:55.856385 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.850084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.30.2.252:38759)
0923 14:45:55.850344 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.850352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.852187 (+ 1835us) server_negotiation.cc:407] Connection header received
0923 14:45:55.852277 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.852280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.852333 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.852412 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:55.852692 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.853350 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.854840 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.855498 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.855580 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.855732 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.856130 (+ 398us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.856236 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260923 14:45:55.856827 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.850026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:38759 (local address 127.0.0.1:34814)
0923 14:45:55.852067 (+ 2041us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.852081 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.852185 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.852447 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.852451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.852471 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:55.852587 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.852592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.853488 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.853492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.854684 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.854698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.855872 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.855875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.855971 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.856004 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.856581 (+ 577us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.856664 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1976,"thread_start_us":63,"threads_started":1}
I20260923 14:45:55.858177 5188 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 14:45:55.858736 5648 sys_catalog.cc:455] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba2e576559541c8b9fe54558d267710. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.859000 5648 sys_catalog.cc:458] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: This master's current role is: LEARNER
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.252@KRBTEST.COM
Sep 23 14:45:55 dist-test-slave-wjf2 krb5kdc[4349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174755, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.253@KRBTEST.COM
I20260923 14:45:55.866747 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.851206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:47213 (local address 127.30.2.252:38759)
0923 14:45:55.853144 (+ 1938us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.853148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.853669 (+ 521us) server_negotiation.cc:407] Connection header received
0923 14:45:55.853832 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.853835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.853880 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.853936 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.856682 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.857457 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.858833 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.859217 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.859294 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.863095 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.863101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.863108 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.863133 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.864958 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.865438 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.865442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.865444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.865532 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.865848 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.865851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.865852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.866071 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.866162 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.866481 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.866592 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1843,"thread_start_us":87,"threads_started":1}
I20260923 14:45:55.867179 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.851045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.195:58751)
0923 14:45:55.851617 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.851680 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.852800 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.853325 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.853336 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.853852 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.854470 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.854485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.855264 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.855267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.856656 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.856669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.857459 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.857464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.857702 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.858383 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.858402 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.860249 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.862360 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.862366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.862381 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.862723 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.863061 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.863064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.863066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.863251 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.864625 (+ 1374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.864632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.864950 (+ 318us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.865461 (+ 511us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.865801 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":145,"threads_started":1}
I20260923 14:45:55.867834 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.851045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:38759 (local address 127.30.2.195:47213)
0923 14:45:55.853595 (+ 2550us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.853608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.853733 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.854054 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.854056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.855992 (+ 1936us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.856343 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.856353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.857804 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.857806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.858699 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.858709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.859672 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.859675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.859819 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.861370 (+ 1551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.861393 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.862938 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.865058 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.865061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.865068 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.865319 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.865633 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.865636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.865638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.865736 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.866220 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.866223 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.866306 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.866994 (+ 688us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.867128 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2449,"mutex_wait_us":128,"thread_start_us":666,"threads_started":1}
I20260923 14:45:55.867949 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.850522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:58751 (local address 127.30.2.254:32951)
0923 14:45:55.850649 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.850654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.851808 (+ 1154us) server_negotiation.cc:407] Connection header received
0923 14:45:55.852985 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.852988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.853032 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.853112 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.854610 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.855154 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.856781 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.857173 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.857260 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.860388 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.860392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.860394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.860420 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.862210 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.862840 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.862845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.862848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.862935 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.863653 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.863657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.863659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.864462 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.864567 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.867640 (+ 3073us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.867762 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:45:55.869237 5631 heartbeater.cc:347] Connected to a master server at 127.30.2.252:38759
I20260923 14:45:55.869237 5632 heartbeater.cc:347] Connected to a master server at 127.30.2.254:32951
I20260923 14:45:55.869506 5631 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:55.870024 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.851045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43049 (local address 127.30.2.195:49023)
0923 14:45:55.853271 (+ 2226us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.853284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.853372 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.853711 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.853714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.860040 (+ 6326us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:55.860175 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.860186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.861896 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.861899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.862766 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.862776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.863497 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.863501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.863592 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.864031 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:55.864049 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:55.865463 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:55.868391 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.868394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.868396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.868647 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.868976 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:55.868979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:55.868981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:55.869066 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:55.869517 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:55.869521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:55.869618 (+ 97us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.869705 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.869850 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2142,"mutex_wait_us":25,"spinlock_wait_cycles":18176,"thread_start_us":141,"threads_started":1}
I20260923 14:45:55.870083 5631 heartbeater.cc:511] Master 127.30.2.252:38759 requested a full tablet report, sending...
I20260923 14:45:55.870151 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.860497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.30.2.252:38759)
0923 14:45:55.860609 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.860614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.860936 (+ 322us) server_negotiation.cc:407] Connection header received
0923 14:45:55.861104 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.861110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.861155 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.861204 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.862222 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.862925 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.864105 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.864445 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.864538 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.866531 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.866537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.866545 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.866582 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.868232 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.868783 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.868786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.868789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.868867 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.869177 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.869181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.869182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.869389 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.869488 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.869901 (+ 413us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.870000 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:45:55.870609 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.850975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:49023 (local address 127.30.2.253:43049)
0923 14:45:55.851096 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.851101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.853370 (+ 2269us) server_negotiation.cc:407] Connection header received
0923 14:45:55.853455 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.853458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.853513 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.853585 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.860317 (+ 6732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.861199 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.862901 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.863252 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.863327 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.865619 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.865624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.865627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.865656 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.868276 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.868777 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.868780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.868782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.868867 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.869184 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.869188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.869190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.869380 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.869469 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.870312 (+ 843us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.870458 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:45:55.871668 5633 heartbeater.cc:347] Connected to a master server at 127.30.2.253:43049
I20260923 14:45:55.871760 5633 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:55.871948 5633 heartbeater.cc:511] Master 127.30.2.253:43049 requested a full tablet report, sending...
I20260923 14:45:55.872155 5178 ts_manager.cc:194] Registered new tserver with Master: 5d3f26ce1e3a472281fbd8391587db1f (127.30.2.195:33897)
I20260923 14:45:55.872644 4991 ts_manager.cc:194] Registered new tserver with Master: 5d3f26ce1e3a472281fbd8391587db1f (127.30.2.195:33897)
I20260923 14:45:55.869506 5632 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:55.873546 5632 heartbeater.cc:511] Master 127.30.2.254:32951 requested a full tablet report, sending...
I20260923 14:45:55.874130 4394 ts_manager.cc:194] Re-registered known tserver with Master: 5d3f26ce1e3a472281fbd8391587db1f (127.30.2.195:33897)
I20260923 14:45:55.874919 4394 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:58751
I20260923 14:45:55.878566 5644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.856199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.30.2.254:32951)
0923 14:45:55.860905 (+ 4706us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.860912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.860926 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:45:55.860962 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.860967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.861008 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.863806 (+ 2798us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.863934 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.871090 (+ 7156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.872426 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.872756 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.873209 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.874453 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.874457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.874460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.874741 (+ 281us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.876775 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.877230 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.877233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.877235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.877320 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.877612 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.877614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.877615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.877843 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.877967 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.878242 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.878376 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4633,"thread_start_us":64,"threads_started":1}
I20260923 14:45:55.878859 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.856370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.30.2.253:43049)
0923 14:45:55.870599 (+ 14229us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.870615 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.870646 (+ 31us) server_negotiation.cc:407] Connection header received
0923 14:45:55.870779 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.870782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.870835 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.870931 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:55.871772 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.872511 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.873621 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.874074 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.874522 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.875387 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:55.875391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:55.875394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:55.875416 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:55.877049 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.877558 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.877565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.877570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.877651 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:55.877941 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:55.877945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:55.877946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:55.878177 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:55.878293 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.878563 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.878660 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14153,"thread_start_us":83,"threads_started":1}
I20260923 14:45:55.881160 4394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:45:55.884155 4394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:55.886813 5521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: NON_VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: true } } }
I20260923 14:45:55.886947 5521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:55.890280 5521 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710: attempting to promote NON_VOTER bceaed5b40c94555bf6695b45659ae09 to VOTER
I20260923 14:45:55.890604 5521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } }
I20260923 14:45:55.892372 5188 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 LEARNER]: Refusing update from remote peer 6ba2e576559541c8b9fe54558d267710: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 14:45:55.892761 5651 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260923 14:45:55.894001 5005 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 FOLLOWER]: Refusing update from remote peer 6ba2e576559541c8b9fe54558d267710: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 14:45:55.894490 5651 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260923 14:45:55.894657 5652 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bceaed5b40c94555bf6695b45659ae09 (127.30.2.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:55.895551 5005 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bceaed5b40c94555bf6695b45659ae09 (127.30.2.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:55.896096 5651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:55.896225 5651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:55.896108 5652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:55.896551 5652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:55.896659 5188 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bceaed5b40c94555bf6695b45659ae09 (127.30.2.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:55.897176 5661 sys_catalog.cc:455] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:55.897312 5661 sys_catalog.cc:458] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:45:55.897660 5652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:55.897755 5652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:55.898751 5648 sys_catalog.cc:455] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ba2e576559541c8b9fe54558d267710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:55.898903 5648 sys_catalog.cc:458] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:45:55.906085 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.900514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33897 (local address 127.0.0.1:44942)
0923 14:45:55.900641 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.900656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.900734 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.901577 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.901583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.901608 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:55.901714 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.901721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.903418 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.903421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.904525 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.904534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.905676 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.905679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.905783 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.905817 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.905870 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.905945 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 14:45:55.906812 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.900709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.30.2.195:33897)
0923 14:45:55.901011 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.901027 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.901052 (+ 25us) server_negotiation.cc:407] Connection header received
0923 14:45:55.901151 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.901157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.901413 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.901896 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:55.902199 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.903300 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.904646 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.905534 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.906361 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.906492 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.906534 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.906623 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260923 14:45:55.906909 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.901181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:42417 (local address 127.0.0.1:56822)
0923 14:45:55.901724 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.901741 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.902000 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.902621 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.902623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.902650 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:55.902733 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.902738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.904035 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.904038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.905230 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.905238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.906510 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.906513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.906600 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.906622 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.906703 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.906766 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":101,"threads_started":1}
I20260923 14:45:55.907315 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.901355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.30.2.193:42417)
0923 14:45:55.902113 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.902128 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.902156 (+ 28us) server_negotiation.cc:407] Connection header received
0923 14:45:55.902253 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.902258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.902469 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.902643 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:55.902849 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.903906 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.905364 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.906185 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.906294 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.906450 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.907069 (+ 619us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.907171 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":91,"threads_started":1}
I20260923 14:45:55.907830 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.901513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41339 (local address 127.0.0.1:56822)
0923 14:45:55.902902 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:55.902915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:55.903010 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:55.903554 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:55.903556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:55.903570 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:55.903643 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.903648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.905048 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.905052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.906185 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:55.906195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.907423 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.907426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:55.907522 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.907539 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:45:55.907630 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 14:45:55.907698 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1321,"thread_start_us":80,"threads_started":1}
I20260923 14:45:55.907925 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:55.901675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.30.2.194:41339)
0923 14:45:55.902012 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:55.902025 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:55.903010 (+ 985us) server_negotiation.cc:407] Connection header received
0923 14:45:55.903130 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:55.903135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:55.903298 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:55.903429 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:55.903756 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.904896 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.906317 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:55.907124 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:55.907234 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:55.907356 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:55.907668 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 14:45:55.907754 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":108,"threads_started":1}
I20260923 14:45:55.910183 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed3c1f969e6432e94094e78f97ff2d2 (DEFAULT_TABLE table=test-table [id=0ae81dab57cc40a2862612c2e3b1b5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:55.910713 5428 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed3c1f969e6432e94094e78f97ff2d2 (DEFAULT_TABLE table=test-table [id=0ae81dab57cc40a2862612c2e3b1b5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:55.911402 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed3c1f969e6432e94094e78f97ff2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:55.911481 5428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed3c1f969e6432e94094e78f97ff2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:55.911126 5291 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed3c1f969e6432e94094e78f97ff2d2 (DEFAULT_TABLE table=test-table [id=0ae81dab57cc40a2862612c2e3b1b5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:55.912056 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed3c1f969e6432e94094e78f97ff2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:55.918279 5667 tablet_bootstrap.cc:492] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8: Bootstrap starting.
I20260923 14:45:55.919334 5669 tablet_bootstrap.cc:492] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f: Bootstrap starting.
I20260923 14:45:55.921012 5667 tablet_bootstrap.cc:654] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:55.921494 5669 tablet_bootstrap.cc:654] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:55.921959 5667 log.cc:824] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:55.922315 5669 log.cc:824] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:55.924011 5668 tablet_bootstrap.cc:492] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a: Bootstrap starting.
I20260923 14:45:55.924533 5669 tablet_bootstrap.cc:492] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f: No bootstrap required, opened a new log
I20260923 14:45:55.924571 5667 tablet_bootstrap.cc:492] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8: No bootstrap required, opened a new log
I20260923 14:45:55.924744 5669 ts_tablet_manager.cc:1397] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:45:55.924742 5667 ts_tablet_manager.cc:1397] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 14:45:55.926779 5668 tablet_bootstrap.cc:654] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:55.928333 5668 log.cc:824] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:55.929409 5669 raft_consensus.cc:348] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:55.929697 5669 raft_consensus.cc:374] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:55.929760 5669 raft_consensus.cc:729] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3f26ce1e3a472281fbd8391587db1f, State: Initialized, Role: FOLLOWER
I20260923 14:45:55.930282 5669 consensus_queue.cc:260] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:55.930398 5667 raft_consensus.cc:348] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:55.930666 5668 tablet_bootstrap.cc:492] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a: No bootstrap required, opened a new log
I20260923 14:45:55.930656 5667 raft_consensus.cc:374] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:55.930716 5667 raft_consensus.cc:729] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f02345dda9e49928bda61615714a6b8, State: Initialized, Role: FOLLOWER
I20260923 14:45:55.930861 5668 ts_tablet_manager.cc:1397] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 14:45:55.931298 5667 consensus_queue.cc:260] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:55.931381 5669 ts_tablet_manager.cc:1428] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260923 14:45:55.932309 5667 ts_tablet_manager.cc:1428] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260923 14:45:55.932507 5495 heartbeater.cc:503] Master 127.30.2.254:32951 was elected leader, sending a full tablet report...
I20260923 14:45:55.932974 5632 heartbeater.cc:503] Master 127.30.2.254:32951 was elected leader, sending a full tablet report...
I20260923 14:45:55.937212 5668 raft_consensus.cc:348] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:55.937446 5668 raft_consensus.cc:374] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:55.937498 5668 raft_consensus.cc:729] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c974d96d963445caf911d5aee86d92a, State: Initialized, Role: FOLLOWER
I20260923 14:45:55.937919 5668 consensus_queue.cc:260] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:55.938846 5668 ts_tablet_manager.cc:1428] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260923 14:45:55.940055 5358 heartbeater.cc:503] Master 127.30.2.254:32951 was elected leader, sending a full tablet report...
I20260923 14:45:56.006286 5675 raft_consensus.cc:482] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:56.006573 5675 raft_consensus.cc:504] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:56.008066 5675 leader_election.cc:288] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d3f26ce1e3a472281fbd8391587db1f (127.30.2.195:33897), 1f02345dda9e49928bda61615714a6b8 (127.30.2.194:41339)
I20260923 14:45:56.012332 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:56.008464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:32783 (local address 127.30.2.194:41339)
0923 14:45:56.008626 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:56.008632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:56.008723 (+ 91us) server_negotiation.cc:407] Connection header received
0923 14:45:56.008875 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:56.008880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:56.008964 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:56.009066 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:56.009399 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.009960 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.011180 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.011677 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.011831 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:56.011921 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:56.012098 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 14:45:56.012192 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:45:56.012342 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:56.008311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41339 (local address 127.30.2.193:32783)
0923 14:45:56.008623 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:56.008638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:56.008754 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:56.009078 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:56.009081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:56.009098 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:56.009275 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:56.009283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.010080 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.010130 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:56.011035 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:56.011045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.011837 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.011843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:56.011961 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:56.011989 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 14:45:56.012067 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:45:56.012157 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":33,"thread_start_us":67,"threads_started":1}
I20260923 14:45:56.012419 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:56.008410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:40529 (local address 127.30.2.195:33897)
0923 14:45:56.008625 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:56.008632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:56.008717 (+ 85us) server_negotiation.cc:407] Connection header received
0923 14:45:56.008915 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:56.008918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:56.008979 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:56.009065 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:56.009501 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.010028 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.011355 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.011818 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.011956 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:56.012006 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:56.012227 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 14:45:56.012300 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260923 14:45:56.012683 5676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:56.008296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33897 (local address 127.30.2.193:40529)
0923 14:45:56.008604 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:56.008622 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:56.008777 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:56.009125 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:56.009129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:56.009148 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:56.009348 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:56.009360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.010247 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.010251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:56.011207 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:56.011218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.011955 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.011959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:56.012080 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:56.012112 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 14:45:56.012198 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 14:45:56.012269 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":96,"threads_started":1}
I20260923 14:45:56.013204 5448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed3c1f969e6432e94094e78f97ff2d2" candidate_uuid: "6c974d96d963445caf911d5aee86d92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f02345dda9e49928bda61615714a6b8" is_pre_election: true
I20260923 14:45:56.013281 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed3c1f969e6432e94094e78f97ff2d2" candidate_uuid: "6c974d96d963445caf911d5aee86d92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3f26ce1e3a472281fbd8391587db1f" is_pre_election: true
I20260923 14:45:56.013527 5448 raft_consensus.cc:2454] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c974d96d963445caf911d5aee86d92a in term 0.
I20260923 14:45:56.013576 5585 raft_consensus.cc:2454] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c974d96d963445caf911d5aee86d92a in term 0.
I20260923 14:45:56.014016 5246 leader_election.cc:302] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f02345dda9e49928bda61615714a6b8, 6c974d96d963445caf911d5aee86d92a; no voters:
I20260923 14:45:56.014338 5675 raft_consensus.cc:2781] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:56.014429 5675 raft_consensus.cc:482] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:56.014467 5675 raft_consensus.cc:3037] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:56.015642 5675 raft_consensus.cc:504] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:56.016112 5675 leader_election.cc:288] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [CANDIDATE]: Term 1 election: Requested vote from peers 5d3f26ce1e3a472281fbd8391587db1f (127.30.2.195:33897), 1f02345dda9e49928bda61615714a6b8 (127.30.2.194:41339)
I20260923 14:45:56.016415 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed3c1f969e6432e94094e78f97ff2d2" candidate_uuid: "6c974d96d963445caf911d5aee86d92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3f26ce1e3a472281fbd8391587db1f"
I20260923 14:45:56.016516 5448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed3c1f969e6432e94094e78f97ff2d2" candidate_uuid: "6c974d96d963445caf911d5aee86d92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f02345dda9e49928bda61615714a6b8"
I20260923 14:45:56.016538 5585 raft_consensus.cc:3037] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:56.016644 5448 raft_consensus.cc:3037] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:56.017891 5585 raft_consensus.cc:2454] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c974d96d963445caf911d5aee86d92a in term 1.
I20260923 14:45:56.017891 5448 raft_consensus.cc:2454] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c974d96d963445caf911d5aee86d92a in term 1.
I20260923 14:45:56.018201 5244 leader_election.cc:302] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3f26ce1e3a472281fbd8391587db1f, 6c974d96d963445caf911d5aee86d92a; no voters:
I20260923 14:45:56.018373 5675 raft_consensus.cc:2781] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:56.018567 5675 raft_consensus.cc:686] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 1 LEADER]: Becoming Leader. State: Replica: 6c974d96d963445caf911d5aee86d92a, State: Running, Role: LEADER
I20260923 14:45:56.018944 5675 consensus_queue.cc:237] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:56.022625 4394 catalog_manager.cc:5495] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c974d96d963445caf911d5aee86d92a (127.30.2.193). New cstate: current_term: 1 leader_uuid: "6c974d96d963445caf911d5aee86d92a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } health_report { overall_health: UNKNOWN } } }
I20260923 14:45:56.059307 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:56.055598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.30.2.193:42417)
0923 14:45:56.055769 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:56.055809 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:56.055828 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:45:56.055998 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:56.056001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:56.056076 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:56.056166 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:56.056389 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.056938 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.057913 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.058150 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.058259 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:56.058500 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:56.058939 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:56.059027 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:56.059091 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 14:45:56.059139 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 14:45:56.065454 5448 raft_consensus.cc:1261] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8 [term 1 FOLLOWER]: Refusing update from remote peer 6c974d96d963445caf911d5aee86d92a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:56.065454 5585 raft_consensus.cc:1261] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f [term 1 FOLLOWER]: Refusing update from remote peer 6c974d96d963445caf911d5aee86d92a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:56.066198 5678 consensus_queue.cc:1035] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:56.066409 5675 consensus_queue.cc:1035] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:56.081081 5682 mvcc.cc:205] Tried to move back new op lower bound from 7332555800836165632 to 7332555800656650240. Current Snapshot: MvccSnapshot[applied={T|T < 7332555800836165632 or (T in {7332555800836165632})}]
I20260923 14:45:56.085381 4393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:56.085549 4393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:56.089924 4393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ba2e576559541c8b9fe54558d267710: Sending DeleteTablet for 3 replicas of tablet 3ed3c1f969e6432e94094e78f97ff2d2
I20260923 14:45:56.090855 5428 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ed3c1f969e6432e94094e78f97ff2d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:56 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:56822
I20260923 14:45:56.091071 5291 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ed3c1f969e6432e94094e78f97ff2d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:56 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:56822
I20260923 14:45:56.091553 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ed3c1f969e6432e94094e78f97ff2d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:56 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:44942
I20260923 14:45:56.092437 5694 tablet_replica.cc:330] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f: stopping tablet replica
I20260923 14:45:56.092470 5692 tablet_replica.cc:330] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8: stopping tablet replica
I20260923 14:45:56.092770 5694 raft_consensus.cc:2229] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:56.092996 5692 raft_consensus.cc:2229] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:56.093159 5694 raft_consensus.cc:2258] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:56.093446 5692 raft_consensus.cc:2258] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:56.093874 5693 tablet_replica.cc:330] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a: stopping tablet replica
I20260923 14:45:56.094141 5693 raft_consensus.cc:2229] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:45:56.094483 5693 raft_consensus.cc:2258] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:56.095340 5692 ts_tablet_manager.cc:1905] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:56.095551 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 4362
I20260923 14:45:56.096074 5693 ts_tablet_manager.cc:1905] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:56.098510 5692 ts_tablet_manager.cc:1918] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:45:56.098665 5692 log.cc:1196] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/3ed3c1f969e6432e94094e78f97ff2d2
I20260923 14:45:56.099057 5692 ts_tablet_manager.cc:1939] T 3ed3c1f969e6432e94094e78f97ff2d2 P 1f02345dda9e49928bda61615714a6b8: Deleting consensus metadata
I20260923 14:45:56.099414 5693 ts_tablet_manager.cc:1918] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:45:56.099521 5693 log.cc:1196] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/3ed3c1f969e6432e94094e78f97ff2d2
I20260923 14:45:56.099869 5693 ts_tablet_manager.cc:1939] T 3ed3c1f969e6432e94094e78f97ff2d2 P 6c974d96d963445caf911d5aee86d92a: Deleting consensus metadata
W20260923 14:45:56.104342 5381 connection.cc:535] server connection from 127.0.0.1:56822 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 14:45:56.104722 5244 connection.cc:535] server connection from 127.0.0.1:56822 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 14:45:56.105557 5694 ts_tablet_manager.cc:1905] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:56.108575 5694 ts_tablet_manager.cc:1918] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:45:56.108671 5694 log.cc:1196] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/3ed3c1f969e6432e94094e78f97ff2d2
I20260923 14:45:56.108997 5694 ts_tablet_manager.cc:1939] T 3ed3c1f969e6432e94094e78f97ff2d2 P 5d3f26ce1e3a472281fbd8391587db1f: Deleting consensus metadata
W20260923 14:45:56.110121 5517 connection.cc:441] server connection from 127.0.0.1:44942 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44942 (request call id 1) could send its response
I20260923 14:45:56.112753 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:56.094252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.30.2.253:43049)
0923 14:45:56.094370 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:56.094374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:56.100347 (+ 5973us) server_negotiation.cc:407] Connection header received
0923 14:45:56.100583 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:56.100587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:56.100647 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:56.100715 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:56.101679 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.102532 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.103813 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.104218 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.104307 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:56.107132 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:56.107137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:56.107140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:56.107168 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:56.109089 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:56.109660 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:56.109664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:56.109666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:56.109761 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:56.110340 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:56.110363 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:56.110368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:56.110622 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:56.110762 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:56.112467 (+ 1705us) server_negotiation.cc:299] Negotiation successful
0923 14:45:56.112596 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 14:45:56.121618 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:56.111020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.30.2.253:43049)
0923 14:45:56.111128 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:56.111133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:56.111145 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:45:56.111180 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:56.111183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:56.111230 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:56.111309 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:56.112153 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.112864 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.114919 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.115261 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.115332 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:56.117813 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:56.117816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:56.117819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:56.117842 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:56.119600 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:56.120173 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:56.120178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:56.120180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:56.120267 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:56.120597 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:56.120601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:56.120603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:56.120821 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:56.120939 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:56.121310 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 14:45:56.121425 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:45:56.122476 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:56.093893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.30.2.252:38759)
0923 14:45:56.094014 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:56.094019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:56.113273 (+ 19254us) server_negotiation.cc:407] Connection header received
0923 14:45:56.113436 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:56.113440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:56.113482 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:56.113528 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:56.114372 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.115079 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.116220 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.116547 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.116656 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:56.118201 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:56.118205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:56.118209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:56.118235 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:56.120407 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:56.120926 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:56.120931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:56.120933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:56.121021 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:56.121349 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:56.121353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:56.121355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:56.121590 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:56.121715 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:56.122188 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 14:45:56.122308 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:45:56.124352 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:56.111018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.30.2.252:38759)
0923 14:45:56.111169 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:56.111174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:56.111538 (+ 364us) server_negotiation.cc:407] Connection header received
0923 14:45:56.111640 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:56.111647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:56.111705 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:56.111835 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:56.112643 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.113398 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.114809 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:56.115172 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:56.115269 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:56.118734 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:56.118739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:56.118745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:56.118772 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:56.120920 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:56.121365 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:56.121368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:56.121370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:56.122018 (+ 648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:56.123467 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:56.123470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:56.123472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:56.123652 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:56.123819 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:56.124078 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 14:45:56.124190 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 14:45:57.079269 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.078819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.195:58317)
0923 14:45:57.079082 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.079175 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
W20260923 14:45:57.079617 5632 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:32951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
I20260923 14:45:57.083511 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.082977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.194:39157)
0923 14:45:57.083263 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.083371 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
W20260923 14:45:57.084008 5495 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:32951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
I20260923 14:45:57.084137 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.083681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.193:46853)
0923 14:45:57.083957 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.084031 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":62,"threads_started":1}
W20260923 14:45:57.084415 5358 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:32951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
I20260923 14:45:57.591025 5706 raft_consensus.cc:482] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ba2e576559541c8b9fe54558d267710)
I20260923 14:45:57.591243 5706 raft_consensus.cc:504] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } }
I20260923 14:45:57.592289 5706 leader_election.cc:288] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ba2e576559541c8b9fe54558d267710 (127.30.2.254:32951), d393dc58431a42f98fc87bdf91dff6c3 (127.30.2.253:43049)
I20260923 14:45:57.592734 5707 raft_consensus.cc:482] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ba2e576559541c8b9fe54558d267710)
I20260923 14:45:57.593071 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.592480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:35430)
0923 14:45:57.592860 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.592946 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":48,"thread_start_us":74,"threads_started":1}
I20260923 14:45:57.592972 5707 raft_consensus.cc:504] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } }
W20260923 14:45:57.593256 5165 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
I20260923 14:45:57.594101 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.593868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:35438)
0923 14:45:57.593993 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.594037 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260923 14:45:57.594201 5165 leader_election.cc:334] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ba2e576559541c8b9fe54558d267710 (127.30.2.254:32951): Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
I20260923 14:45:57.594614 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.594076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:35444)
0923 14:45:57.594462 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.594530 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":68,"threads_started":1}
I20260923 14:45:57.594586 5707 leader_election.cc:288] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ba2e576559541c8b9fe54558d267710 (127.30.2.254:32951), bceaed5b40c94555bf6695b45659ae09 (127.30.2.252:38759)
W20260923 14:45:57.594846 4982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
I20260923 14:45:57.595661 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.595454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:35454)
0923 14:45:57.595557 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.595604 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 14:45:57.595774 4982 leader_election.cc:334] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ba2e576559541c8b9fe54558d267710 (127.30.2.254:32951): Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
I20260923 14:45:57.597059 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.592476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43049 (local address 127.0.0.1:45734)
0923 14:45:57.592836 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.592854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:57.592962 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:57.593367 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:57.593370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:57.593388 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:57.593495 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.593502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.594517 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.594520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.595873 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.595886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.596657 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.596660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.596751 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.596769 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 14:45:57.596862 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 14:45:57.596943 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":90,"threads_started":1}
I20260923 14:45:57.597096 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.592632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.30.2.253:43049)
0923 14:45:57.592861 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:57.592867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:57.592915 (+ 48us) server_negotiation.cc:407] Connection header received
0923 14:45:57.593086 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:57.593092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:57.593152 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:57.593243 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:57.593628 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.594387 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.595992 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.596500 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.596615 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.596675 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:57.596878 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 14:45:57.596997 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":44,"threads_started":1}
I20260923 14:45:57.597656 5005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bceaed5b40c94555bf6695b45659ae09" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d393dc58431a42f98fc87bdf91dff6c3" is_pre_election: true
I20260923 14:45:57.597844 5005 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceaed5b40c94555bf6695b45659ae09 in term 1.
I20260923 14:45:57.598176 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.594442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.30.2.252:38759)
0923 14:45:57.594761 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:57.594769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:57.594780 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:45:57.594816 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:57.594819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:57.594855 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:57.594923 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:57.595292 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.596072 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.597100 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.597654 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.597767 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.597818 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:57.597997 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 14:45:57.598062 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":260,"thread_start_us":89,"threads_started":1}
I20260923 14:45:57.598189 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.594159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:38759 (local address 127.0.0.1:34850)
0923 14:45:57.594466 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.594480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:57.594564 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:57.595056 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:57.595061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:57.595081 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:57.595168 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.595174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.596181 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.596187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.596971 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.596979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.597771 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.597775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.597867 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.597889 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:45:57.597983 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 14:45:57.598051 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":42,"threads_started":1}
I20260923 14:45:57.598213 5162 leader_election.cc:302] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceaed5b40c94555bf6695b45659ae09, d393dc58431a42f98fc87bdf91dff6c3; no voters: 6ba2e576559541c8b9fe54558d267710
I20260923 14:45:57.598460 5706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 14:45:57.598541 5706 raft_consensus.cc:482] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ba2e576559541c8b9fe54558d267710)
I20260923 14:45:57.598601 5706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:45:57.598692 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d393dc58431a42f98fc87bdf91dff6c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bceaed5b40c94555bf6695b45659ae09" is_pre_election: true
I20260923 14:45:57.599745 5706 raft_consensus.cc:504] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } }
I20260923 14:45:57.600051 5188 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d393dc58431a42f98fc87bdf91dff6c3 in current term 2: Already voted for candidate bceaed5b40c94555bf6695b45659ae09 in this term.
I20260923 14:45:57.600183 5706 leader_election.cc:288] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [CANDIDATE]: Term 2 election: Requested vote from peers 6ba2e576559541c8b9fe54558d267710 (127.30.2.254:32951), d393dc58431a42f98fc87bdf91dff6c3 (127.30.2.253:43049)
I20260923 14:45:57.600498 5005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bceaed5b40c94555bf6695b45659ae09" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d393dc58431a42f98fc87bdf91dff6c3"
I20260923 14:45:57.600615 5005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:45:57.600931 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.600288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:35460)
0923 14:45:57.600819 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.600869 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450}
I20260923 14:45:57.600943 4981 leader_election.cc:302] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d393dc58431a42f98fc87bdf91dff6c3; no voters: 6ba2e576559541c8b9fe54558d267710, bceaed5b40c94555bf6695b45659ae09
I20260923 14:45:57.601496 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.601319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:35470)
0923 14:45:57.601408 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.601451 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 14:45:57.601624 5165 leader_election.cc:334] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6ba2e576559541c8b9fe54558d267710 (127.30.2.254:32951): Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
I20260923 14:45:57.601895 5005 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceaed5b40c94555bf6695b45659ae09 in term 2.
I20260923 14:45:57.602003 5707 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:45:57.602174 5162 leader_election.cc:302] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceaed5b40c94555bf6695b45659ae09, d393dc58431a42f98fc87bdf91dff6c3; no voters: 6ba2e576559541c8b9fe54558d267710
I20260923 14:45:57.602331 5706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 14:45:57.602510 5706 raft_consensus.cc:686] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [term 2 LEADER]: Becoming Leader. State: Replica: bceaed5b40c94555bf6695b45659ae09, State: Running, Role: LEADER
I20260923 14:45:57.602751 5706 consensus_queue.cc:237] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } }
I20260923 14:45:57.603911 5716 sys_catalog.cc:455] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bceaed5b40c94555bf6695b45659ae09. Latest consensus state: current_term: 2 leader_uuid: "bceaed5b40c94555bf6695b45659ae09" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:57.604027 5716 sys_catalog.cc:458] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:57.604331 5718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:57.606614 5718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:57.607084 5718 catalog_manager.cc:1259] Loaded cluster ID: ea6a320ed7bf45599adce879d795567e
I20260923 14:45:57.607127 5718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:57.607652 5718 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:57.608100 5718 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: Loaded TSK: 0
I20260923 14:45:57.608702 5718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:45:57.639174 5178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:45:57.642414 5178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:45:57.645678 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.645266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:35490)
0923 14:45:57.645404 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.645582 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 14:45:57.645694 5005 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [term 2 FOLLOWER]: Refusing update from remote peer bceaed5b40c94555bf6695b45659ae09: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260923 14:45:57.646234 5716 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 14:45:57.646512 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.646287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.0.0.1:35500)
0923 14:45:57.646396 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.646446 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 14:45:57.646705 5165 consensus_peers.cc:487] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 -> Peer 6ba2e576559541c8b9fe54558d267710 (127.30.2.254:32951): Couldn't send request to peer 6ba2e576559541c8b9fe54558d267710. Status: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 14:45:57.647769 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bceaed5b40c94555bf6695b45659ae09. Latest consensus state: current_term: 2 leader_uuid: "bceaed5b40c94555bf6695b45659ae09" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:57.647940 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:45:57.649163 5716 sys_catalog.cc:455] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "bceaed5b40c94555bf6695b45659ae09" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:57.649273 5716 sys_catalog.cc:458] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09 [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:57.649580 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "bceaed5b40c94555bf6695b45659ae09" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6ba2e576559541c8b9fe54558d267710" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 32951 } } peers { permanent_uuid: "d393dc58431a42f98fc87bdf91dff6c3" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43049 } attrs { promote: false } } peers { permanent_uuid: "bceaed5b40c94555bf6695b45659ae09" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 38759 } attrs { promote: false } } }
I20260923 14:45:57.649739 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P d393dc58431a42f98fc87bdf91dff6c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:45:57.660076 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.655203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33897 (local address 127.0.0.1:44952)
0923 14:45:57.655446 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.655458 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:57.655525 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:57.656866 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:57.656870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:57.656888 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:57.656960 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.656965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.657833 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.657840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.658671 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.658683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.659549 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.659553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.659659 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.659730 (+ 71us) client_negotiation.cc:769] Sending connection context
0923 14:45:57.659827 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 14:45:57.659911 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":48}
I20260923 14:45:57.660135 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.655460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41339 (local address 127.0.0.1:56826)
0923 14:45:57.655724 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.655739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:57.655946 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:57.656233 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:57.656236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:57.656253 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:57.656352 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.656357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.657228 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.657234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.658430 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.658440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.659736 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.659739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.659859 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.659879 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:45:57.659946 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 14:45:57.660008 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":102,"threads_started":1}
I20260923 14:45:57.660192 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.655122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.30.2.193:42417)
0923 14:45:57.655997 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:57.656002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:57.656016 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:45:57.656069 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:57.656072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:57.656122 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:57.656500 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:57.656617 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.657372 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.658782 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.659393 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.659514 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.659583 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:57.659999 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 14:45:57.660071 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":75,"threads_started":1}
I20260923 14:45:57.660190 5726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.655596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.30.2.194:41339)
0923 14:45:57.655864 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:57.655871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:57.655888 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:45:57.656065 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:57.656069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:57.656129 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:57.656214 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:57.656466 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.657130 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.658704 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.659616 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.659730 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.659826 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:57.659977 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 14:45:57.660050 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260923 14:45:57.660727 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.654983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:42417 (local address 127.0.0.1:56840)
0923 14:45:57.655140 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.655154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:57.655238 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:57.656258 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:57.656261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:57.656279 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:57.656388 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.656395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.657490 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.657493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.658626 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.658633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.659515 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.659518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.659621 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.659723 (+ 102us) client_negotiation.cc:769] Sending connection context
0923 14:45:57.659817 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 14:45:57.659897 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":50}
I20260923 14:45:57.660738 5428 tablet_service.cc:1467] Processing CreateTablet for tablet 2420b2739d114b5b8cb66871ce71bb6e (DEFAULT_TABLE table=test-table [id=c7ae1cc59e58484c95ec84f3762faaa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:57.661067 5428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2420b2739d114b5b8cb66871ce71bb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:57.661218 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.655688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.30.2.195:33897)
0923 14:45:57.656617 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:57.656622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:57.656658 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:45:57.656706 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:57.656709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:57.656752 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:57.656839 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:57.657074 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.657730 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.658810 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.659417 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.660896 (+ 1479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.660965 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:57.660999 (+ 34us) server_negotiation.cc:299] Negotiation successful
0923 14:45:57.661074 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":107,"threads_started":1}
I20260923 14:45:57.661294 5291 tablet_service.cc:1467] Processing CreateTablet for tablet 2420b2739d114b5b8cb66871ce71bb6e (DEFAULT_TABLE table=test-table [id=c7ae1cc59e58484c95ec84f3762faaa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:57.661578 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2420b2739d114b5b8cb66871ce71bb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:57.661722 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 2420b2739d114b5b8cb66871ce71bb6e (DEFAULT_TABLE table=test-table [id=c7ae1cc59e58484c95ec84f3762faaa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:45:57.662005 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2420b2739d114b5b8cb66871ce71bb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:57.663713 5728 tablet_bootstrap.cc:492] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8: Bootstrap starting.
I20260923 14:45:57.663995 5729 tablet_bootstrap.cc:492] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f: Bootstrap starting.
I20260923 14:45:57.664777 5728 tablet_bootstrap.cc:654] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:57.664903 5729 tablet_bootstrap.cc:654] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:57.665470 5730 tablet_bootstrap.cc:492] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a: Bootstrap starting.
I20260923 14:45:57.665916 5728 tablet_bootstrap.cc:492] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8: No bootstrap required, opened a new log
I20260923 14:45:57.665989 5728 ts_tablet_manager.cc:1397] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 14:45:57.666334 5728 raft_consensus.cc:348] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:57.666420 5728 raft_consensus.cc:374] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:57.666442 5728 raft_consensus.cc:729] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f02345dda9e49928bda61615714a6b8, State: Initialized, Role: FOLLOWER
I20260923 14:45:57.666530 5728 consensus_queue.cc:260] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:57.666867 5728 ts_tablet_manager.cc:1428] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:45:57.667876 5730 tablet_bootstrap.cc:654] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:57.669086 5730 tablet_bootstrap.cc:492] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a: No bootstrap required, opened a new log
I20260923 14:45:57.669185 5730 ts_tablet_manager.cc:1397] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260923 14:45:57.669324 5494 heartbeater.cc:503] Master 127.30.2.252:38759 was elected leader, sending a full tablet report...
I20260923 14:45:57.669387 5729 tablet_bootstrap.cc:492] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f: No bootstrap required, opened a new log
I20260923 14:45:57.669445 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.667148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.194:51617)
0923 14:45:57.669262 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.669323 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":88,"threads_started":1}
I20260923 14:45:57.669468 5729 ts_tablet_manager.cc:1397] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260923 14:45:57.669647 5730 raft_consensus.cc:348] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:57.669757 5730 raft_consensus.cc:374] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:57.669795 5730 raft_consensus.cc:729] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c974d96d963445caf911d5aee86d92a, State: Initialized, Role: FOLLOWER
I20260923 14:45:57.669934 5730 consensus_queue.cc:260] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:57.669968 5729 raft_consensus.cc:348] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:57.670079 5729 raft_consensus.cc:374] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:57.670113 5729 raft_consensus.cc:729] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3f26ce1e3a472281fbd8391587db1f, State: Initialized, Role: FOLLOWER
I20260923 14:45:57.670230 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.669839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.194:38289)
0923 14:45:57.670102 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.670153 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
I20260923 14:45:57.670269 5730 ts_tablet_manager.cc:1428] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 14:45:57.670293 5729 consensus_queue.cc:260] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:57.670985 5729 ts_tablet_manager.cc:1428] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 14:45:57.671044 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.670673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.193:35401)
0923 14:45:57.670904 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.670955 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260923 14:45:57.672474 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.672257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.193:40395)
0923 14:45:57.672372 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.672416 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 14:45:57.672492 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.672082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.195:55039)
0923 14:45:57.672336 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.672390 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
I20260923 14:45:57.673182 5631 heartbeater.cc:503] Master 127.30.2.252:38759 was elected leader, sending a full tablet report...
I20260923 14:45:57.673228 5357 heartbeater.cc:503] Master 127.30.2.252:38759 was elected leader, sending a full tablet report...
I20260923 14:45:57.675549 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.674198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.195:42271)
0923 14:45:57.675434 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.675485 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1159}
W20260923 14:45:57.687723 5360 tablet.cc:2369] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:45:57.851828 5634 tablet.cc:2369] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:45:57.890393 5738 raft_consensus.cc:482] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:45:57.890559 5738 raft_consensus.cc:504] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:57.891669 5738 leader_election.cc:288] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c974d96d963445caf911d5aee86d92a (127.30.2.193:42417), 1f02345dda9e49928bda61615714a6b8 (127.30.2.194:41339)
I20260923 14:45:57.895928 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.892047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:37445 (local address 127.30.2.193:42417)
0923 14:45:57.892220 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:57.892226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:57.892274 (+ 48us) server_negotiation.cc:407] Connection header received
0923 14:45:57.892336 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:57.892339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:57.892394 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:57.892482 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:57.892891 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.893540 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.894856 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.895281 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.895377 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.895418 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:57.895679 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 14:45:57.895795 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 14:45:57.895931 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.891951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:42417 (local address 127.30.2.195:37445)
0923 14:45:57.892083 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.892099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:57.892210 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:57.892581 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:57.892587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:57.892608 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:57.892758 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.892766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.893672 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.893676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.894711 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.894722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.895432 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.895439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.895536 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.895563 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 14:45:57.895651 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 14:45:57.895721 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":38}
I20260923 14:45:57.896046 5726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.892408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:50473 (local address 127.30.2.194:41339)
0923 14:45:57.892550 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:57.892556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:57.892717 (+ 161us) server_negotiation.cc:407] Connection header received
0923 14:45:57.892797 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:57.892803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:57.892863 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:57.892929 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:45:57.893247 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.893846 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.894922 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.895372 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.895510 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.895578 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:57.895832 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 14:45:57.895914 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 14:45:57.896071 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.892271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41339 (local address 127.30.2.195:50473)
0923 14:45:57.892623 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.892635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:57.892718 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:57.892967 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:57.892971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:57.892988 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:45:57.893089 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.893096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.893969 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.893972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.894755 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:57.894764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.895555 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.895559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:57.895668 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.895693 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 14:45:57.895766 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 14:45:57.895854 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":24,"thread_start_us":144,"threads_started":1}
I20260923 14:45:57.896467 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b2739d114b5b8cb66871ce71bb6e" candidate_uuid: "5d3f26ce1e3a472281fbd8391587db1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c974d96d963445caf911d5aee86d92a" is_pre_election: true
I20260923 14:45:57.896538 5448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b2739d114b5b8cb66871ce71bb6e" candidate_uuid: "5d3f26ce1e3a472281fbd8391587db1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f02345dda9e49928bda61615714a6b8" is_pre_election: true
I20260923 14:45:57.896755 5311 raft_consensus.cc:2454] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3f26ce1e3a472281fbd8391587db1f in term 0.
I20260923 14:45:57.896765 5448 raft_consensus.cc:2454] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3f26ce1e3a472281fbd8391587db1f in term 0.
I20260923 14:45:57.897182 5519 leader_election.cc:302] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f02345dda9e49928bda61615714a6b8, 5d3f26ce1e3a472281fbd8391587db1f; no voters:
I20260923 14:45:57.897461 5738 raft_consensus.cc:2781] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:45:57.897532 5738 raft_consensus.cc:482] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:45:57.897593 5738 raft_consensus.cc:3037] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:57.898602 5738 raft_consensus.cc:504] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:57.898929 5738 leader_election.cc:288] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [CANDIDATE]: Term 1 election: Requested vote from peers 6c974d96d963445caf911d5aee86d92a (127.30.2.193:42417), 1f02345dda9e49928bda61615714a6b8 (127.30.2.194:41339)
I20260923 14:45:57.899410 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b2739d114b5b8cb66871ce71bb6e" candidate_uuid: "5d3f26ce1e3a472281fbd8391587db1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c974d96d963445caf911d5aee86d92a"
I20260923 14:45:57.899457 5448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b2739d114b5b8cb66871ce71bb6e" candidate_uuid: "5d3f26ce1e3a472281fbd8391587db1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f02345dda9e49928bda61615714a6b8"
I20260923 14:45:57.899533 5311 raft_consensus.cc:3037] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a [term 0 FOLLOWER]: Advancing to term 1
W20260923 14:45:57.899534 5497 tablet.cc:2369] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:45:57.899618 5448 raft_consensus.cc:3037] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:57.900645 5448 raft_consensus.cc:2454] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3f26ce1e3a472281fbd8391587db1f in term 1.
I20260923 14:45:57.900651 5311 raft_consensus.cc:2454] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3f26ce1e3a472281fbd8391587db1f in term 1.
I20260923 14:45:57.900928 5519 leader_election.cc:302] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f02345dda9e49928bda61615714a6b8, 5d3f26ce1e3a472281fbd8391587db1f; no voters:
I20260923 14:45:57.901144 5738 raft_consensus.cc:2781] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:57.901369 5738 raft_consensus.cc:686] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 1 LEADER]: Becoming Leader. State: Replica: 5d3f26ce1e3a472281fbd8391587db1f, State: Running, Role: LEADER
I20260923 14:45:57.901614 5738 consensus_queue.cc:237] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } } peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } }
I20260923 14:45:57.902117 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.901860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.195:35313)
0923 14:45:57.901985 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.902059 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 14:45:57.904376 5177 catalog_manager.cc:5495] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d3f26ce1e3a472281fbd8391587db1f (127.30.2.195). New cstate: current_term: 1 leader_uuid: "5d3f26ce1e3a472281fbd8391587db1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d3f26ce1e3a472281fbd8391587db1f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 } health_report { overall_health: UNKNOWN } } }
I20260923 14:45:57.954588 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.951302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.30.2.195:33897)
0923 14:45:57.951434 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:57.951438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:57.951452 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:45:57.951559 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:57.951561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:57.951618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:57.951715 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:45:57.951984 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.952474 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.953258 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:57.953461 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:57.953537 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:57.953779 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:45:57.954210 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:45:57.954295 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:57.954436 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 14:45:57.954485 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:45:57.957506 5448 raft_consensus.cc:1261] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8 [term 1 FOLLOWER]: Refusing update from remote peer 5d3f26ce1e3a472281fbd8391587db1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:57.957793 5311 raft_consensus.cc:1261] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a [term 1 FOLLOWER]: Refusing update from remote peer 5d3f26ce1e3a472281fbd8391587db1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:45:57.958132 5742 consensus_queue.cc:1035] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f02345dda9e49928bda61615714a6b8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:57.958325 5738 consensus_queue.cc:1035] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c974d96d963445caf911d5aee86d92a" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:45:57.961488 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.960503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.194:40589)
0923 14:45:57.961300 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.961377 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":716}
I20260923 14:45:57.962086 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.961814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.193:44539)
0923 14:45:57.961943 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.962005 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 14:45:57.962998 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.962704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.195:35199)
0923 14:45:57.962867 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.962925 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 14:45:57.962324 5746 mvcc.cc:205] Tried to move back new op lower bound from 7332555808588795904 to 7332555808368001024. Current Snapshot: MvccSnapshot[applied={T|T < 7332555808588795904}]
I20260923 14:45:57.964082 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.963799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.193:49691)
0923 14:45:57.963947 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.964002 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 14:45:57.965257 5748 mvcc.cc:205] Tried to move back new op lower bound from 7332555808588795904 to 7332555808368001024. Current Snapshot: MvccSnapshot[applied={T|T < 7332555808588795904 or (T in {7332555808588795904})}]
I20260923 14:45:57.967012 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.966785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.195:35469)
0923 14:45:57.966903 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.966954 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 14:45:57.969157 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.968937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:32951 (local address 127.30.2.194:51559)
0923 14:45:57.969034 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.969088 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:32951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 14:45:57.973810 5178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:57.973979 5178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:45:57.978015 5178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bceaed5b40c94555bf6695b45659ae09: Sending DeleteTablet for 3 replicas of tablet 2420b2739d114b5b8cb66871ce71bb6e
I20260923 14:45:57.978691 5291 tablet_service.cc:1514] Processing DeleteTablet for tablet 2420b2739d114b5b8cb66871ce71bb6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:57 UTC) from {username='oryx', principal='oryx/127.30.2.252@KRBTEST.COM'} at 127.0.0.1:56840
I20260923 14:45:57.978714 5428 tablet_service.cc:1514] Processing DeleteTablet for tablet 2420b2739d114b5b8cb66871ce71bb6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:57 UTC) from {username='oryx', principal='oryx/127.30.2.252@KRBTEST.COM'} at 127.0.0.1:56826
I20260923 14:45:57.979358 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 2420b2739d114b5b8cb66871ce71bb6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:45:57 UTC) from {username='oryx', principal='oryx/127.30.2.252@KRBTEST.COM'} at 127.0.0.1:44952
I20260923 14:45:57.979421 5757 tablet_replica.cc:330] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a: stopping tablet replica
I20260923 14:45:57.979566 5757 raft_consensus.cc:2229] T 2420b2739d114b5b8cb66871ce71bb6e P 6c974d96d963445caf911d5aee86d92a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:57.979583 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 5139
I20260923 14:45:57.981503 5758 tablet_replica.cc:330] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f: stopping tablet replica
I20260923 14:45:57.981652 5758 raft_consensus.cc:2229] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:45:57.981953 5758 raft_consensus.cc:2258] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:57.982558 5758 ts_tablet_manager.cc:1905] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:57.984699 5758 ts_tablet_manager.cc:1918] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:45:57.984794 5758 log.cc:1196] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/2420b2739d114b5b8cb66871ce71bb6e
I20260923 14:45:57.985057 5758 ts_tablet_manager.cc:1939] T 2420b2739d114b5b8cb66871ce71bb6e P 5d3f26ce1e3a472281fbd8391587db1f: Deleting consensus metadata
I20260923 14:45:57.985848 5162 catalog_manager.cc:4841] TS 5d3f26ce1e3a472281fbd8391587db1f (127.30.2.195:33897): tablet 2420b2739d114b5b8cb66871ce71bb6e (table test-table [id=c7ae1cc59e58484c95ec84f3762faaa4]) successfully deleted
I20260923 14:45:57.987972 5756 tablet_replica.cc:330] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8: stopping tablet replica
I20260923 14:45:57.988123 5756 raft_consensus.cc:2229] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:45:57.988296 5756 raft_consensus.cc:2258] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:45:57.988953 5756 ts_tablet_manager.cc:1905] T 2420b2739d114b5b8cb66871ce71bb6e P 1f02345dda9e49928bda61615714a6b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:45:57.990034 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.989739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:42417 (local address 127.0.0.1:56852)
0923 14:45:57.989896 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.989946 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:42417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 14:45:57.990252 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 5365
W20260923 14:45:57.992604 5164 catalog_manager.cc:4584] TS 6c974d96d963445caf911d5aee86d92a (127.30.2.193:42417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2420b2739d114b5b8cb66871ce71bb6e: Network error: Client connection negotiation failed: client connection to 127.30.2.193:42417: connect: Connection refused (error 111)
I20260923 14:45:57.999262 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 5501
I20260923 14:45:57.999295 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:57.999020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:41339 (local address 127.0.0.1:56838)
0923 14:45:57.999150 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:57.999207 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.194:41339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 14:45:58.004750 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 4858
I20260923 14:45:58.011093 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 4967
2026-09-23T14:45:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5538 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:45:58 dist-test-slave-wjf2 krb5kdc[5761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:45:58 dist-test-slave-wjf2 krb5kdc[5761](info): set up 2 sockets
Sep 23 14:45:58 dist-test-slave-wjf2 krb5kdc[5761](info): commencing operation
krb5kdc: starting...
W20260923 14:45:59.085709 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:45:59 dist-test-slave-wjf2 krb5kdc[5761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:45:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.254.keytab.
Entry for principal oryx/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.254.keytab.
I20260923 14:45:59.207991 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:40839
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.254.keytab
--principal=oryx/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:59.320616 5774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:59.320922 5774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:59.320998 5774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:59.325058 5774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:45:59.325133 5774 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 14:45:59.325163 5774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:59.325207 5774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:59.325241 5774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:45:59.325295 5774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:45:59.330430 5774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:40839
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.254.keytab
--principal=oryx/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:59.331485 5774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:59.332564 5774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:59.341107 5779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:59.341202 5774 server_base.cc:1034] running on GCE node
W20260923 14:45:59.341117 5782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:59.341161 5780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:59.341992 5774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:59.343076 5774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:59.344301 5774 hybrid_clock.cc:648] HybridClock initialized: now 1790174759344268 us; error 43 us; skew 500 ppm
Sep 23 14:45:59 dist-test-slave-wjf2 krb5kdc[5761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:59.346990 5774 init.cc:377] Logged in from keytab as oryx/127.30.2.254@KRBTEST.COM (short username oryx)
I20260923 14:45:59.347874 5774 webserver.cc:460] Webserver started at http://127.30.2.254:36843/ using document root <none> and password file <none>
I20260923 14:45:59.348387 5774 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:59.348443 5774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:59.348599 5774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:59.350490 5774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "326417a82b8f4a629d88c9f665647d6f"
format_stamp: "Formatted at 2026-09-23 14:45:59 on dist-test-slave-wjf2"
server_key: "a6ca0094f1493697e366aa91c2ff5f5f"
server_key_iv: "8fe4b23e1efb7ffaf91e7deddbbd7b1e"
server_key_version: "encryptionkey@0"
I20260923 14:45:59.350980 5774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "326417a82b8f4a629d88c9f665647d6f"
format_stamp: "Formatted at 2026-09-23 14:45:59 on dist-test-slave-wjf2"
server_key: "a6ca0094f1493697e366aa91c2ff5f5f"
server_key_iv: "8fe4b23e1efb7ffaf91e7deddbbd7b1e"
server_key_version: "encryptionkey@0"
I20260923 14:45:59.354573 5774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260923 14:45:59.356946 5789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:59.358173 5774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:45:59.358292 5774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "326417a82b8f4a629d88c9f665647d6f"
format_stamp: "Formatted at 2026-09-23 14:45:59 on dist-test-slave-wjf2"
server_key: "a6ca0094f1493697e366aa91c2ff5f5f"
server_key_iv: "8fe4b23e1efb7ffaf91e7deddbbd7b1e"
server_key_version: "encryptionkey@0"
I20260923 14:45:59.358389 5774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:59.379464 5774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:59.382928 5774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:59.383153 5774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:59.391177 5774 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:40839
I20260923 14:45:59.391227 5851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:40839 every 8 connection(s)
I20260923 14:45:59.392210 5774 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:45:59.394232 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 5774
I20260923 14:45:59.394364 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:45:59.394685 30731 external_mini_cluster.cc:1428] Setting key 8ce02abedb631cbdc94c80bbe8d57575
I20260923 14:45:59.395331 5852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:45:59.402148 5852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f: Bootstrap starting.
Sep 23 14:45:59 dist-test-slave-wjf2 krb5kdc[5761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:45:59.404919 5852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f: Neither blocks nor log segments found. Creating new log.
I20260923 14:45:59.405951 5852 log.cc:824] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:45:59.407769 5852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f: No bootstrap required, opened a new log
I20260923 14:45:59.409664 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:59.396166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.30.2.254:40839)
0923 14:45:59.396573 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:59.396602 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:59.396640 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:45:59.397319 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:59.397337 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:59.397695 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:59.397976 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:59.398875 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.399984 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.401181 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.401669 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.401792 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:59.404735 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:59.404740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:59.404753 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:59.404781 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:59.406672 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:59.407173 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:59.407180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:59.407184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:59.407298 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:59.407607 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:59.407611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:59.407613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:59.408125 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:59.408337 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:59.408678 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 14:45:59.408963 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":134,"threads_started":1}
I20260923 14:45:59.410868 5852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326417a82b8f4a629d88c9f665647d6f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40839 } }
I20260923 14:45:59.411113 5852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:45:59.411197 5852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326417a82b8f4a629d88c9f665647d6f, State: Initialized, Role: FOLLOWER
I20260923 14:45:59.411625 5852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326417a82b8f4a629d88c9f665647d6f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40839 } }
I20260923 14:45:59.411775 5852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:45:59.411891 5852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:45:59.412011 5852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:45:59.412849 5852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326417a82b8f4a629d88c9f665647d6f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40839 } }
I20260923 14:45:59.413182 5852 leader_election.cc:302] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 326417a82b8f4a629d88c9f665647d6f; no voters:
I20260923 14:45:59.413455 5852 leader_election.cc:288] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:45:59.413571 5857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:45:59.413827 5857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [term 1 LEADER]: Becoming Leader. State: Replica: 326417a82b8f4a629d88c9f665647d6f, State: Running, Role: LEADER
I20260923 14:45:59.414158 5857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326417a82b8f4a629d88c9f665647d6f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40839 } }
I20260923 14:45:59.414690 5852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:45:59.416007 5858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "326417a82b8f4a629d88c9f665647d6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326417a82b8f4a629d88c9f665647d6f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40839 } } }
I20260923 14:45:59.415982 5859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 326417a82b8f4a629d88c9f665647d6f. Latest consensus state: current_term: 1 leader_uuid: "326417a82b8f4a629d88c9f665647d6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326417a82b8f4a629d88c9f665647d6f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40839 } } }
I20260923 14:45:59.416112 5859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:59.416112 5858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f [sys.catalog]: This master's current role is: LEADER
I20260923 14:45:59.417181 5866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:45:59.420548 5866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:45:59.426430 5866 catalog_manager.cc:1347] Generated new cluster ID: fce5714f36104de69181f536db11cbd8
I20260923 14:45:59.426491 5866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:45:59.444623 5866 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:45:59.445843 5866 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:45:59.454430 5866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f: Generated new TSK 0
I20260923 14:45:59.455152 5866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab.
Entry for principal oryx/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab.
I20260923 14:45:59.491856 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40839
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab
--principal=oryx/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:59.602941 5878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:59.603152 5878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:59.603191 5878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:59.607149 5878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:59.607201 5878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:59.607273 5878 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:45:59.607345 5878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:45:59.612412 5878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab
--principal=oryx/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.2.254:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:59.613368 5878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:59.614395 5878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:59.624135 5883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:59.624182 5886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:59.624161 5884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:59.624378 5878 server_base.cc:1034] running on GCE node
I20260923 14:45:59.624780 5878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:59.625387 5878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:59.626591 5878 hybrid_clock.cc:648] HybridClock initialized: now 1790174759626545 us; error 53 us; skew 500 ppm
Sep 23 14:45:59 dist-test-slave-wjf2 krb5kdc[5761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:59.629272 5878 init.cc:377] Logged in from keytab as oryx/127.30.2.193@KRBTEST.COM (short username oryx)
I20260923 14:45:59.630155 5878 webserver.cc:460] Webserver started at http://127.30.2.193:41823/ using document root <none> and password file <none>
I20260923 14:45:59.630709 5878 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:59.630795 5878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:59.631019 5878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:59.633056 5878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "b3cc10078fed4fdb901dccca9ee53a1d"
format_stamp: "Formatted at 2026-09-23 14:45:59 on dist-test-slave-wjf2"
server_key: "857748a6a6c370da56acedfd04987fad"
server_key_iv: "a95a7006ecfd3f33daaccf51d3496317"
server_key_version: "encryptionkey@0"
I20260923 14:45:59.633602 5878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "b3cc10078fed4fdb901dccca9ee53a1d"
format_stamp: "Formatted at 2026-09-23 14:45:59 on dist-test-slave-wjf2"
server_key: "857748a6a6c370da56acedfd04987fad"
server_key_iv: "a95a7006ecfd3f33daaccf51d3496317"
server_key_version: "encryptionkey@0"
I20260923 14:45:59.637319 5878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:45:59.639657 5893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:59.641441 5878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:45:59.641563 5878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "b3cc10078fed4fdb901dccca9ee53a1d"
format_stamp: "Formatted at 2026-09-23 14:45:59 on dist-test-slave-wjf2"
server_key: "857748a6a6c370da56acedfd04987fad"
server_key_iv: "a95a7006ecfd3f33daaccf51d3496317"
server_key_version: "encryptionkey@0"
I20260923 14:45:59.641651 5878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:59.653988 5878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:59.656736 5878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:59.656879 5878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:59.658286 5878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:59.658334 5878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:59.658398 5878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:59.658432 5878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:45:59 dist-test-slave-wjf2 krb5kdc[5761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:45:59.669281 5878 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:35403
I20260923 14:45:59.669308 6008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:35403 every 8 connection(s)
I20260923 14:45:59.670118 5878 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:45:59.672549 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:59.659238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:59939 (local address 127.30.2.254:40839)
0923 14:45:59.659369 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:59.659380 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:59.660463 (+ 1083us) server_negotiation.cc:407] Connection header received
0923 14:45:59.661668 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:59.661671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:59.661733 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:59.661816 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:59.663423 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.664361 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.665676 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.666085 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.666202 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:59.669141 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:59.669146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:59.669148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:59.669177 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:59.670805 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:59.671295 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:59.671298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:59.671299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:59.671376 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:59.671622 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:59.671623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:59.671625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:59.671841 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:59.671926 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:59.672328 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 14:45:59.672433 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:45:59.673295 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:59.659829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40839 (local address 127.30.2.193:59939)
0923 14:45:59.660316 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:59.660358 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:59.661445 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:59.662046 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:59.662056 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:59.662531 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:59.663239 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:59.663260 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.664512 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.664516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:59.665544 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:59.665555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.666252 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.666257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:59.666467 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:59.667066 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:59.667089 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:59.668919 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:59.670941 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:59.670946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:59.670959 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:59.671216 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:59.671453 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:59.671455 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:59.671456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:59.671567 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:59.671922 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:59.671926 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:59.672129 (+ 203us) client_negotiation.cc:769] Sending connection context
0923 14:45:59.672277 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 14:45:59.672509 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":134,"threads_started":1}
I20260923 14:45:59.674522 6009 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40839
I20260923 14:45:59.674688 6009 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:59.675218 6009 heartbeater.cc:511] Master 127.30.2.254:40839 requested a full tablet report, sending...
I20260923 14:45:59.676858 5805 ts_manager.cc:194] Registered new tserver with Master: b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403)
I20260923 14:45:59.677616 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 5878
I20260923 14:45:59.677762 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:45:59.678103 30731 external_mini_cluster.cc:1428] Setting key af5d628c8ce95af07c86c7d72eb25587
I20260923 14:45:59.678313 5805 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:59939
I20260923 14:45:59.685086 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:59.676689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:33305 (local address 127.30.2.254:40839)
0923 14:45:59.676817 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:59.676821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:59.676987 (+ 166us) server_negotiation.cc:407] Connection header received
0923 14:45:59.677095 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:59.677097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:59.677140 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:59.677204 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:59.677700 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.678432 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.680237 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.680631 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.680755 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:59.681968 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:59.681971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:59.681973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:59.681994 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:59.683346 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:59.683919 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:59.683922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:59.683923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:59.683993 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:59.684318 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:59.684321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:59.684322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:59.684469 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:59.684543 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:59.684891 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 14:45:59.684980 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:45:59.685295 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:59.676626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40839 (local address 127.30.2.193:33305)
0923 14:45:59.676902 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:59.676919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:59.677016 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:59.677250 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:59.677253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:59.677449 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:59.677563 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:59.677570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.678560 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.678563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:59.680095 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:59.680105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.680749 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.680751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:59.680852 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:59.681343 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:59.681362 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:59.681830 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:59.683475 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:59.683482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:59.683487 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:59.683803 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:59.684110 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:59.684114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:59.684116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:59.684221 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:59.684572 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:59.684578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:59.684708 (+ 130us) client_negotiation.cc:769] Sending connection context
0923 14:45:59.684844 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 14:45:59.685010 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for oryx/127.30.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab.
Entry for principal oryx/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab.
I20260923 14:45:59.706748 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40839
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab
--principal=oryx/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:45:59.818349 6018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:45:59.818614 6018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:45:59.818696 6018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:45:59.822638 6018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:45:59.822688 6018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:45:59.822726 6018 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:45:59.822790 6018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:45:59.827909 6018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab
--principal=oryx/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.2.254:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:45:59.828930 6018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:45:59.829958 6018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:45:59.839680 6026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:59.839672 6023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:45:59.839660 6024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:45:59.839993 6018 server_base.cc:1034] running on GCE node
I20260923 14:45:59.840423 6018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:45:59.841097 6018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:45:59.842296 6018 hybrid_clock.cc:648] HybridClock initialized: now 1790174759842267 us; error 50 us; skew 500 ppm
Sep 23 14:45:59 dist-test-slave-wjf2 krb5kdc[5761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:45:59.845150 6018 init.cc:377] Logged in from keytab as oryx/127.30.2.194@KRBTEST.COM (short username oryx)
I20260923 14:45:59.846078 6018 webserver.cc:460] Webserver started at http://127.30.2.194:45071/ using document root <none> and password file <none>
I20260923 14:45:59.846652 6018 fs_manager.cc:359] Metadata directory not provided
I20260923 14:45:59.846746 6018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:45:59.847002 6018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:45:59.849236 6018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "771221bf1acd40d5a569a5b01cc76989"
format_stamp: "Formatted at 2026-09-23 14:45:59 on dist-test-slave-wjf2"
server_key: "9b47501a38538532da7fd6efa3eb9674"
server_key_iv: "d973ddeea2f09498b08bc42e5ff0238c"
server_key_version: "encryptionkey@0"
I20260923 14:45:59.849884 6018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "771221bf1acd40d5a569a5b01cc76989"
format_stamp: "Formatted at 2026-09-23 14:45:59 on dist-test-slave-wjf2"
server_key: "9b47501a38538532da7fd6efa3eb9674"
server_key_iv: "d973ddeea2f09498b08bc42e5ff0238c"
server_key_version: "encryptionkey@0"
I20260923 14:45:59.853677 6018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:45:59.856124 6033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:59.857326 6018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:45:59.857466 6018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "771221bf1acd40d5a569a5b01cc76989"
format_stamp: "Formatted at 2026-09-23 14:45:59 on dist-test-slave-wjf2"
server_key: "9b47501a38538532da7fd6efa3eb9674"
server_key_iv: "d973ddeea2f09498b08bc42e5ff0238c"
server_key_version: "encryptionkey@0"
I20260923 14:45:59.857575 6018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:45:59.896603 6018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:45:59.901113 6018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:45:59.901356 6018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:45:59.903750 6018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:45:59.903847 6018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:45:59.903918 6018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:45:59.903942 6018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:45:59 dist-test-slave-wjf2 krb5kdc[5761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:45:59.917601 6018 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:44001
I20260923 14:45:59.918727 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:59.904536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40839 (local address 127.30.2.194:39771)
0923 14:45:59.905023 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:59.905071 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:59.905967 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:59.906526 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:59.906533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:59.906950 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:59.907537 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:59.907556 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.908699 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.908706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:59.909979 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:59.909989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.910642 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.910646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:59.910860 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:59.911476 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:59.911501 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:59.913889 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:59.915979 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:59.915984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:59.915999 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:59.916313 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:59.916614 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:59.916617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:59.916619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:59.916819 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:59.917258 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:59.917265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:59.917545 (+ 280us) client_negotiation.cc:769] Sending connection context
0923 14:45:59.917743 (+ 198us) client_negotiation.cc:240] Negotiation successful
0923 14:45:59.917997 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":158,"threads_started":1}
I20260923 14:45:59.918953 6018 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:45:59.919618 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:59.904010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:39771 (local address 127.30.2.254:40839)
0923 14:45:59.904170 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:59.904175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:59.905165 (+ 990us) server_negotiation.cc:407] Connection header received
0923 14:45:59.906181 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:59.906184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:59.906254 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:59.906353 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:59.907719 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.908568 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.910106 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.910471 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.910585 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:59.914059 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:59.914063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:59.914067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:59.914101 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:59.915861 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:59.916415 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:59.916421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:59.916423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:59.916516 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:59.916944 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:59.916949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:59.916951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:59.917142 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:59.917258 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:59.919365 (+ 2107us) server_negotiation.cc:299] Negotiation successful
0923 14:45:59.919457 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 14:45:59.923094 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 6018
I20260923 14:45:59.923250 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:45:59.923619 30731 external_mini_cluster.cc:1428] Setting key b16d7a301279af18f055fcc589c1bc5e
I20260923 14:45:59.925029 6148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:44001 every 8 connection(s)
I20260923 14:45:59.926000 6149 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40839
I20260923 14:45:59.926185 6149 heartbeater.cc:464] Registering TS with master...
I20260923 14:45:59.926735 6149 heartbeater.cc:511] Master 127.30.2.254:40839 requested a full tablet report, sending...
I20260923 14:45:59.927933 5806 ts_manager.cc:194] Registered new tserver with Master: 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001)
I20260923 14:45:59.928745 5806 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:39771
I20260923 14:45:59.936875 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:59.923624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:60507 (local address 127.30.2.254:40839)
0923 14:45:59.923803 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 14:45:59.923810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:45:59.925942 (+ 2132us) server_negotiation.cc:407] Connection header received
0923 14:45:59.926122 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:45:59.926125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:45:59.926168 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:45:59.926232 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:45:59.926858 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.927699 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.929031 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.929363 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.929445 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:59.932673 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:45:59.932683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:45:59.932686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:45:59.932707 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:45:59.934547 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:59.935156 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:59.935162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:59.935165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:59.935263 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:45:59.935700 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:45:59.935705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:45:59.935710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:45:59.936046 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:45:59.936168 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 14:45:59.936555 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 14:45:59.936721 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:45:59.936904 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:45:59.923510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40839 (local address 127.30.2.194:60507)
0923 14:45:59.925870 (+ 2360us) negotiation.cc:107] Waiting for socket to connect
0923 14:45:59.925885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:45:59.926006 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:45:59.926371 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:45:59.926374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:45:59.926577 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:45:59.926704 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:59.926712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.927909 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.927913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:59.928887 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:45:59.928898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:45:59.929593 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:45:59.929598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:45:59.929701 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:45:59.931917 (+ 2216us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:45:59.931938 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:45:59.932502 (+ 564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:45:59.934701 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:59.934708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:59.934712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:59.935016 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:59.935372 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:45:59.935379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:45:59.935383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:45:59.935569 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:45:59.936158 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:45:59.936164 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:45:59.936290 (+ 126us) client_negotiation.cc:769] Sending connection context
0923 14:45:59.936415 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 14:45:59.936607 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2275,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for oryx/127.30.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab.
Entry for principal oryx/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab.
I20260923 14:45:59.959362 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40839
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab
--principal=oryx/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:46:00.073081 6158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:00.073333 6158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:00.073375 6158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:00.077670 6158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:00.077726 6158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:00.077765 6158 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:46:00.077827 6158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:46:00.083230 6158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab
--principal=oryx/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.2.254:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:00.084291 6158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:00.085386 6158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:00.094985 6164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:00.094985 6166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:00.095045 6163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:00.095068 6158 server_base.cc:1034] running on GCE node
I20260923 14:46:00.095690 6158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:00.096336 6158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:00.097540 6158 hybrid_clock.cc:648] HybridClock initialized: now 1790174760097518 us; error 30 us; skew 500 ppm
Sep 23 14:46:00 dist-test-slave-wjf2 krb5kdc[5761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174760, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:00.100636 6158 init.cc:377] Logged in from keytab as oryx/127.30.2.195@KRBTEST.COM (short username oryx)
I20260923 14:46:00.101534 6158 webserver.cc:460] Webserver started at http://127.30.2.195:45703/ using document root <none> and password file <none>
I20260923 14:46:00.102140 6158 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:00.102233 6158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:00.102483 6158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:00.104768 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "2057c7e00aa74c0f855e7dbf4e8db42a"
format_stamp: "Formatted at 2026-09-23 14:46:00 on dist-test-slave-wjf2"
server_key: "83b28fa5f59aa37a56d9148905f29957"
server_key_iv: "dc21a2305f72fef8d7053664c47b106c"
server_key_version: "encryptionkey@0"
I20260923 14:46:00.105371 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "2057c7e00aa74c0f855e7dbf4e8db42a"
format_stamp: "Formatted at 2026-09-23 14:46:00 on dist-test-slave-wjf2"
server_key: "83b28fa5f59aa37a56d9148905f29957"
server_key_iv: "dc21a2305f72fef8d7053664c47b106c"
server_key_version: "encryptionkey@0"
I20260923 14:46:00.109303 6158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:46:00.111840 6173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:00.113113 6158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:46:00.113252 6158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "2057c7e00aa74c0f855e7dbf4e8db42a"
format_stamp: "Formatted at 2026-09-23 14:46:00 on dist-test-slave-wjf2"
server_key: "83b28fa5f59aa37a56d9148905f29957"
server_key_iv: "dc21a2305f72fef8d7053664c47b106c"
server_key_version: "encryptionkey@0"
I20260923 14:46:00.113368 6158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:00.126346 6158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:00.129802 6158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:00.130010 6158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:00.131522 6158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:00.131608 6158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:00.131688 6158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:00.131737 6158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:46:00 dist-test-slave-wjf2 krb5kdc[5761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174760, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:46:00.143391 6158 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:41809
I20260923 14:46:00.143491 6288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:41809 every 8 connection(s)
I20260923 14:46:00.144429 6158 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:46:00.145571 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 6158
I20260923 14:46:00.145741 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:46:00.146133 30731 external_mini_cluster.cc:1428] Setting key a998a58fdfb089507cf33ea32fd8b37d
I20260923 14:46:00.146868 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.132329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:41091 (local address 127.30.2.254:40839)
0923 14:46:00.132519 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.132527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.133308 (+ 781us) server_negotiation.cc:407] Connection header received
0923 14:46:00.134319 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.134324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.134394 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.134507 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:00.136067 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.137076 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.138650 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.138972 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.139083 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.142742 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:00.142745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:00.142748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:00.142773 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:00.144412 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:00.145036 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:00.145039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:00.145041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:00.145144 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:00.145529 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:00.145534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:00.145535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:00.145845 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:00.145965 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.146517 (+ 552us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.146661 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 14:46:00.147572 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.132688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40839 (local address 127.30.2.195:41091)
0923 14:46:00.133126 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:00.133183 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:00.134064 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:00.134749 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:00.134758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:00.135230 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:00.135872 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.135889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.137222 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.137225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.138513 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.138523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.139121 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.139125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.139301 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.139900 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:00.139925 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:00.142589 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:00.144542 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:00.144548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:00.144562 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:00.144907 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:00.145252 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:00.145255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:00.145257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:00.145412 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:00.145980 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:00.145986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:00.146243 (+ 257us) client_negotiation.cc:769] Sending connection context
0923 14:46:00.146424 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 14:46:00.146731 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":123,"threads_started":1}
I20260923 14:46:00.148944 6289 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40839
I20260923 14:46:00.149114 6289 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:00.149657 6289 heartbeater.cc:511] Master 127.30.2.254:40839 requested a full tablet report, sending...
I20260923 14:46:00.150731 5805 ts_manager.cc:194] Registered new tserver with Master: 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195:41809)
I20260923 14:46:00.151611 5805 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:41091
I20260923 14:46:00.157805 6295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.151011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40839 (local address 127.30.2.195:44637)
0923 14:46:00.151304 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:00.151315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:00.151390 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:00.151595 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:00.151597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:00.151854 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:00.152006 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.152013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.152914 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.152917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.153760 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.153769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.154222 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.154224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.154288 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.154753 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:00.154773 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:00.155061 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:00.156581 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:00.156583 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:00.156585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:00.156767 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:00.157019 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:00.157021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:00.157023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:00.157091 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:00.157417 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:00.157419 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:00.157487 (+ 68us) client_negotiation.cc:769] Sending connection context
0923 14:46:00.157565 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:46:00.157673 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":99,"threads_started":1}
I20260923 14:46:00.157838 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.151068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:44637 (local address 127.30.2.254:40839)
0923 14:46:00.151175 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.151179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.151389 (+ 210us) server_negotiation.cc:407] Connection header received
0923 14:46:00.151450 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.151453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.151496 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.151586 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:00.152119 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.152739 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.153877 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.154121 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.154212 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.155174 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:00.155177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:00.155179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:00.155196 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:00.156483 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:00.156860 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:00.156863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:00.156864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:00.156938 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:00.157171 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:00.157174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:00.157175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:00.157336 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:00.157416 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.157644 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.157731 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:46:00.160219 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 14:46:00 dist-test-slave-wjf2 krb5kdc[5761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 14:46:00 dist-test-slave-wjf2 krb5kdc[5761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:00 dist-test-slave-wjf2 krb5kdc[5761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 14:46:00.184832 5855 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:00.178488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.30.2.254:40839)
0923 14:46:00.178634 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.178638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.178650 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:46:00.178692 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.178694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.178732 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.178800 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:00.179588 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.180171 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.181005 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.181214 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.181300 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.184627 (+ 3327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52418: BlockingRecv error: recv got EOF from 127.0.0.1:52418 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 14:46:00 dist-test-slave-wjf2 krb5kdc[5761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:46:00.194710 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.186290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.30.2.254:40839)
0923 14:46:00.186446 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.186451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.186593 (+ 142us) server_negotiation.cc:407] Connection header received
0923 14:46:00.186749 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.186752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.186813 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.186905 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:00.187546 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.188166 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.189007 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.189241 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.189361 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.191615 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:00.191618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:00.191621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:00.191644 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:00.192959 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:00.193405 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:00.193411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:00.193413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:00.193497 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:00.193839 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:00.193843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:00.193845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:00.194034 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:00.194179 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.194432 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.194546 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 14:46:00.197503 5805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:46:00.200688 5805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:46:00.214903 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.209088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:41809 (local address 127.0.0.1:45026)
0923 14:46:00.209645 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:00.209658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:00.210249 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:00.210987 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:00.210989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:00.211005 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:00.211097 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.211106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.212233 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.212237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.213393 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.213403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.214335 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.214339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.214456 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.214497 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 14:46:00.214619 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 14:46:00.214738 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":4,"thread_start_us":102,"threads_started":1}
I20260923 14:46:00.215061 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.210090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.30.2.195:41809)
0923 14:46:00.210542 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.210566 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.210585 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:46:00.210659 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.210662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.210844 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.211238 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:00.211368 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.212123 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.213521 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.214196 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.214309 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.214467 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.214666 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.214814 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":127,"threads_started":1}
I20260923 14:46:00.215488 6313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.209119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.30.2.194:44001)
0923 14:46:00.209819 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.209834 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.209859 (+ 25us) server_negotiation.cc:407] Connection header received
0923 14:46:00.210461 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.210467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.210685 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.210860 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:00.211385 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.212518 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.213908 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.214542 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.214649 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.215230 (+ 581us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.215278 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.215350 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":114,"threads_started":1}
I20260923 14:46:00.215727 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.209836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.30.2.193:35403)
0923 14:46:00.210205 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.210213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.210466 (+ 253us) server_negotiation.cc:407] Connection header received
0923 14:46:00.210547 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.210551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.210702 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.210860 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:00.211157 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.212309 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.213861 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.214697 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.214837 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.214971 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.215506 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.215605 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":118,"threads_started":1}
I20260923 14:46:00.215708 6311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.208967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:44001 (local address 127.0.0.1:42530)
0923 14:46:00.209455 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:00.209509 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:00.210167 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:00.211173 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:00.211179 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:00.211198 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:00.211269 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.211274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.212634 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.212637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.213814 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.213825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.214860 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.214863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.214956 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.214980 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 14:46:00.215049 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 14:46:00.215117 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":54,"thread_start_us":66,"threads_started":1}
I20260923 14:46:00.215709 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.209676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:35403 (local address 127.0.0.1:42180)
0923 14:46:00.210357 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:00.210373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:00.210461 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:00.210865 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:00.210871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:00.210897 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:00.211037 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.211043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.212440 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.212443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.213732 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.213743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.215254 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.215258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.215349 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.215374 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 14:46:00.215454 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 14:46:00.215520 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":18,"thread_start_us":85,"threads_started":1}
I20260923 14:46:00.218245 6223 tablet_service.cc:1467] Processing CreateTablet for tablet eef82b0e3b08400ab2997771964d39ca (DEFAULT_TABLE table=test-table [id=e8d12a8289ce4dcda79031274b63e4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:00.218394 6083 tablet_service.cc:1467] Processing CreateTablet for tablet eef82b0e3b08400ab2997771964d39ca (DEFAULT_TABLE table=test-table [id=e8d12a8289ce4dcda79031274b63e4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:00.218544 5943 tablet_service.cc:1467] Processing CreateTablet for tablet eef82b0e3b08400ab2997771964d39ca (DEFAULT_TABLE table=test-table [id=e8d12a8289ce4dcda79031274b63e4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:00.219269 6083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef82b0e3b08400ab2997771964d39ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:00.219298 6223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef82b0e3b08400ab2997771964d39ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:00.219334 5943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef82b0e3b08400ab2997771964d39ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:00.226042 6317 tablet_bootstrap.cc:492] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: Bootstrap starting.
I20260923 14:46:00.227082 6318 tablet_bootstrap.cc:492] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: Bootstrap starting.
I20260923 14:46:00.227324 6319 tablet_bootstrap.cc:492] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: Bootstrap starting.
I20260923 14:46:00.228641 6317 tablet_bootstrap.cc:654] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:00.229254 6318 tablet_bootstrap.cc:654] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:00.229254 6319 tablet_bootstrap.cc:654] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:00.229460 6317 log.cc:824] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: Log is configured to *not* fsync() on all Append() calls
I20260923 14:46:00.230142 6319 log.cc:824] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:46:00.230510 6318 log.cc:824] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: Log is configured to *not* fsync() on all Append() calls
I20260923 14:46:00.231379 6317 tablet_bootstrap.cc:492] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: No bootstrap required, opened a new log
I20260923 14:46:00.231583 6317 ts_tablet_manager.cc:1397] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:46:00.232100 6319 tablet_bootstrap.cc:492] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: No bootstrap required, opened a new log
I20260923 14:46:00.232281 6319 ts_tablet_manager.cc:1397] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:46:00.232455 6318 tablet_bootstrap.cc:492] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: No bootstrap required, opened a new log
I20260923 14:46:00.232640 6318 ts_tablet_manager.cc:1397] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:46:00.235244 6319 raft_consensus.cc:348] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } }
I20260923 14:46:00.235518 6319 raft_consensus.cc:374] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:00.235574 6319 raft_consensus.cc:729] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2057c7e00aa74c0f855e7dbf4e8db42a, State: Initialized, Role: FOLLOWER
I20260923 14:46:00.235585 6317 raft_consensus.cc:348] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } }
I20260923 14:46:00.235855 6317 raft_consensus.cc:374] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:00.235918 6317 raft_consensus.cc:729] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771221bf1acd40d5a569a5b01cc76989, State: Initialized, Role: FOLLOWER
I20260923 14:46:00.236088 6319 consensus_queue.cc:260] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } }
I20260923 14:46:00.236377 6317 consensus_queue.cc:260] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } }
I20260923 14:46:00.236567 6318 raft_consensus.cc:348] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } }
I20260923 14:46:00.236845 6318 raft_consensus.cc:374] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:00.236901 6318 raft_consensus.cc:729] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cc10078fed4fdb901dccca9ee53a1d, State: Initialized, Role: FOLLOWER
I20260923 14:46:00.237206 6319 ts_tablet_manager.cc:1428] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:46:00.237298 6149 heartbeater.cc:503] Master 127.30.2.254:40839 was elected leader, sending a full tablet report...
I20260923 14:46:00.237322 6318 consensus_queue.cc:260] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } }
I20260923 14:46:00.237403 6317 ts_tablet_manager.cc:1428] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 14:46:00.237536 6289 heartbeater.cc:503] Master 127.30.2.254:40839 was elected leader, sending a full tablet report...
I20260923 14:46:00.238075 6318 ts_tablet_manager.cc:1428] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 14:46:00.238539 6009 heartbeater.cc:503] Master 127.30.2.254:40839 was elected leader, sending a full tablet report...
W20260923 14:46:00.395898 6290 tablet.cc:2369] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:46:00.421664 6010 tablet.cc:2369] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:46:00.426132 6150 tablet.cc:2369] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:46:00.565454 6324 raft_consensus.cc:482] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:46:00.565676 6324 raft_consensus.cc:504] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } }
I20260923 14:46:00.566828 6324 leader_election.cc:288] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195:41809), b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403)
I20260923 14:46:00.571132 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.567233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:46503 (local address 127.30.2.195:41809)
0923 14:46:00.567370 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.567373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.567523 (+ 150us) server_negotiation.cc:407] Connection header received
0923 14:46:00.567723 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.567727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.567822 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.567878 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:00.568359 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.568943 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.570168 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.570562 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.570698 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.570753 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.570923 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.570997 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:46:00.571169 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.567722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:46187 (local address 127.30.2.193:35403)
0923 14:46:00.567891 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.567895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.567935 (+ 40us) server_negotiation.cc:407] Connection header received
0923 14:46:00.568083 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.568087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.568136 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.568210 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:00.568496 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.568985 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.570140 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.570574 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.570707 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.570760 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.570973 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.571054 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 14:46:00.571163 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.567155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:41809 (local address 127.30.2.194:46503)
0923 14:46:00.567416 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:00.567431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:00.567553 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:00.568075 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:00.568078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:00.568099 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:00.568213 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.568220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.569059 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.569063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.570014 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.570024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.570686 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.570689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.570786 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.570818 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 14:46:00.570879 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 14:46:00.570972 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":35,"thread_start_us":102,"threads_started":1}
I20260923 14:46:00.571170 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.567562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:35403 (local address 127.30.2.194:46187)
0923 14:46:00.567846 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:00.567859 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:00.567970 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:00.568231 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:00.568233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:00.568247 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:00.568376 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.568383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.569156 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.569159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.570007 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.570016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.570719 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.570723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.570825 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.570842 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:46:00.570938 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 14:46:00.571003 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":20,"thread_start_us":81,"threads_started":1}
I20260923 14:46:00.571962 6242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef82b0e3b08400ab2997771964d39ca" candidate_uuid: "771221bf1acd40d5a569a5b01cc76989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" is_pre_election: true
I20260923 14:46:00.571966 5963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef82b0e3b08400ab2997771964d39ca" candidate_uuid: "771221bf1acd40d5a569a5b01cc76989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" is_pre_election: true
I20260923 14:46:00.572283 5963 raft_consensus.cc:2454] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771221bf1acd40d5a569a5b01cc76989 in term 0.
I20260923 14:46:00.572283 6242 raft_consensus.cc:2454] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771221bf1acd40d5a569a5b01cc76989 in term 0.
I20260923 14:46:00.572775 6037 leader_election.cc:302] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2057c7e00aa74c0f855e7dbf4e8db42a, 771221bf1acd40d5a569a5b01cc76989; no voters:
I20260923 14:46:00.573015 6324 raft_consensus.cc:2781] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:46:00.573099 6324 raft_consensus.cc:482] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:46:00.573184 6324 raft_consensus.cc:3037] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:00.574319 6324 raft_consensus.cc:504] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } }
I20260923 14:46:00.574666 6324 leader_election.cc:288] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [CANDIDATE]: Term 1 election: Requested vote from peers 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195:41809), b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403)
I20260923 14:46:00.575049 5963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef82b0e3b08400ab2997771964d39ca" candidate_uuid: "771221bf1acd40d5a569a5b01cc76989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cc10078fed4fdb901dccca9ee53a1d"
I20260923 14:46:00.575129 6242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef82b0e3b08400ab2997771964d39ca" candidate_uuid: "771221bf1acd40d5a569a5b01cc76989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a"
I20260923 14:46:00.575184 5963 raft_consensus.cc:3037] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:00.575265 6242 raft_consensus.cc:3037] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:00.576392 5963 raft_consensus.cc:2454] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771221bf1acd40d5a569a5b01cc76989 in term 1.
I20260923 14:46:00.576709 6037 leader_election.cc:302] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771221bf1acd40d5a569a5b01cc76989, b3cc10078fed4fdb901dccca9ee53a1d; no voters:
I20260923 14:46:00.576862 6324 raft_consensus.cc:2781] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:46:00.576958 6242 raft_consensus.cc:2454] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771221bf1acd40d5a569a5b01cc76989 in term 1.
I20260923 14:46:00.577050 6324 raft_consensus.cc:686] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 1 LEADER]: Becoming Leader. State: Replica: 771221bf1acd40d5a569a5b01cc76989, State: Running, Role: LEADER
I20260923 14:46:00.577381 6324 consensus_queue.cc:237] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } }
I20260923 14:46:00.580372 5805 catalog_manager.cc:5495] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 reported cstate change: term changed from 0 to 1, leader changed from <none> to 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194). New cstate: current_term: 1 leader_uuid: "771221bf1acd40d5a569a5b01cc76989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } health_report { overall_health: UNKNOWN } } }
I20260923 14:46:00.675160 6313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.671729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.30.2.194:44001)
0923 14:46:00.671878 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.671883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.671901 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:46:00.671996 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.671998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.672054 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.672164 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:46:00.672406 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.672910 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.673756 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.673969 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.674075 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.674380 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:46:00.674767 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:46:00.674886 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.674964 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.675018 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 14:46:00.681282 5963 raft_consensus.cc:1261] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Refusing update from remote peer 771221bf1acd40d5a569a5b01cc76989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:46:00.681372 6242 raft_consensus.cc:1261] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Refusing update from remote peer 771221bf1acd40d5a569a5b01cc76989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:46:00.682065 6328 consensus_queue.cc:1035] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:00.682308 6324 consensus_queue.cc:1035] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:00.689321 6331 mvcc.cc:205] Tried to move back new op lower bound from 7332555819743088640 to 7332555819327959040. Current Snapshot: MvccSnapshot[applied={T|T < 7332555819743088640}]
I20260923 14:46:00.690465 6333 mvcc.cc:205] Tried to move back new op lower bound from 7332555819743088640 to 7332555819327959040. Current Snapshot: MvccSnapshot[applied={T|T < 7332555819743088640}]
I20260923 14:46:00.690681 6332 mvcc.cc:205] Tried to move back new op lower bound from 7332555819743088640 to 7332555819327959040. Current Snapshot: MvccSnapshot[applied={T|T < 7332555819743088640}]
I20260923 14:46:00.697328 5806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:46:00.697499 5806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:46:00.699915 5806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f: Sending DeleteTablet for 3 replicas of tablet eef82b0e3b08400ab2997771964d39ca
I20260923 14:46:00.700644 6083 tablet_service.cc:1514] Processing DeleteTablet for tablet eef82b0e3b08400ab2997771964d39ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:00 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42530
I20260923 14:46:00.700790 5943 tablet_service.cc:1514] Processing DeleteTablet for tablet eef82b0e3b08400ab2997771964d39ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:00 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42180
I20260923 14:46:00.700783 6223 tablet_service.cc:1514] Processing DeleteTablet for tablet eef82b0e3b08400ab2997771964d39ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:00 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:45026
I20260923 14:46:00.701327 6340 tablet_replica.cc:330] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: stopping tablet replica
I20260923 14:46:00.701360 6342 tablet_replica.cc:330] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: stopping tablet replica
I20260923 14:46:00.701439 5806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 14:46:00.701618 6340 raft_consensus.cc:2229] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:46:00.701623 6342 raft_consensus.cc:2229] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260923 14:46:00.701819 5806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:46:00.701889 6342 raft_consensus.cc:2258] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:00.701949 6340 raft_consensus.cc:2258] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:00.703295 6342 ts_tablet_manager.cc:1905] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:00.703295 6340 ts_tablet_manager.cc:1905] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:00.706164 6342 ts_tablet_manager.cc:1918] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:46:00.706205 6340 ts_tablet_manager.cc:1918] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:46:00.706260 6342 log.cc:1196] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/eef82b0e3b08400ab2997771964d39ca
I20260923 14:46:00.706291 6340 log.cc:1196] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/eef82b0e3b08400ab2997771964d39ca
I20260923 14:46:00.706588 6340 ts_tablet_manager.cc:1939] T eef82b0e3b08400ab2997771964d39ca P 771221bf1acd40d5a569a5b01cc76989: Deleting consensus metadata
I20260923 14:46:00.706920 6342 ts_tablet_manager.cc:1939] T eef82b0e3b08400ab2997771964d39ca P b3cc10078fed4fdb901dccca9ee53a1d: Deleting consensus metadata
I20260923 14:46:00.707773 6341 tablet_replica.cc:330] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: stopping tablet replica
I20260923 14:46:00.708077 6341 raft_consensus.cc:2229] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:46:00.708124 6223 tablet_service.cc:1467] Processing CreateTablet for tablet bdc404b023fa4edbbbde3b87db19fdc2 (DEFAULT_TABLE table=test-table [id=00152946fa5547a0a0db38ce13553a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:00.708344 6341 raft_consensus.cc:2258] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:00.708334 6223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc404b023fa4edbbbde3b87db19fdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:00.708796 5943 tablet_service.cc:1467] Processing CreateTablet for tablet bdc404b023fa4edbbbde3b87db19fdc2 (DEFAULT_TABLE table=test-table [id=00152946fa5547a0a0db38ce13553a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:00.709017 5943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc404b023fa4edbbbde3b87db19fdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:00.710271 6319 tablet_bootstrap.cc:492] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a: Bootstrap starting.
I20260923 14:46:00.711032 6319 tablet_bootstrap.cc:654] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:00.711089 6318 tablet_bootstrap.cc:492] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d: Bootstrap starting.
I20260923 14:46:00.711215 5793 catalog_manager.cc:4841] TS b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403): tablet eef82b0e3b08400ab2997771964d39ca (table test-table [id=e8d12a8289ce4dcda79031274b63e4f6]) successfully deleted
I20260923 14:46:00.711553 5790 catalog_manager.cc:4841] TS 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001): tablet eef82b0e3b08400ab2997771964d39ca (table test-table [id=e8d12a8289ce4dcda79031274b63e4f6]) successfully deleted
I20260923 14:46:00.712028 6318 tablet_bootstrap.cc:654] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:00.712009 6083 tablet_service.cc:1467] Processing CreateTablet for tablet bdc404b023fa4edbbbde3b87db19fdc2 (DEFAULT_TABLE table=test-table [id=00152946fa5547a0a0db38ce13553a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:00.712229 6083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc404b023fa4edbbbde3b87db19fdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:00.712723 6319 tablet_bootstrap.cc:492] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a: No bootstrap required, opened a new log
I20260923 14:46:00.712803 6319 ts_tablet_manager.cc:1397] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 14:46:00.713172 6318 tablet_bootstrap.cc:492] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d: No bootstrap required, opened a new log
I20260923 14:46:00.713243 6318 ts_tablet_manager.cc:1397] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 14:46:00.713253 6319 raft_consensus.cc:348] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.713357 6319 raft_consensus.cc:374] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:00.713394 6319 raft_consensus.cc:729] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2057c7e00aa74c0f855e7dbf4e8db42a, State: Initialized, Role: FOLLOWER
I20260923 14:46:00.713517 6319 consensus_queue.cc:260] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.713673 6318 raft_consensus.cc:348] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.713768 6318 raft_consensus.cc:374] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:00.713781 6319 ts_tablet_manager.cc:1428] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:46:00.713802 6318 raft_consensus.cc:729] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cc10078fed4fdb901dccca9ee53a1d, State: Initialized, Role: FOLLOWER
I20260923 14:46:00.713912 6318 consensus_queue.cc:260] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.714157 6318 ts_tablet_manager.cc:1428] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:46:00.714314 6317 tablet_bootstrap.cc:492] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989: Bootstrap starting.
I20260923 14:46:00.715173 6317 tablet_bootstrap.cc:654] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:00.716473 6317 tablet_bootstrap.cc:492] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989: No bootstrap required, opened a new log
I20260923 14:46:00.716547 6317 ts_tablet_manager.cc:1397] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 14:46:00.716974 6317 raft_consensus.cc:348] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.717087 6317 raft_consensus.cc:374] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:00.717119 6317 raft_consensus.cc:729] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771221bf1acd40d5a569a5b01cc76989, State: Initialized, Role: FOLLOWER
I20260923 14:46:00.717233 6317 consensus_queue.cc:260] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.717451 6317 ts_tablet_manager.cc:1428] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:46:00.717962 6341 ts_tablet_manager.cc:1905] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:00.720304 6341 ts_tablet_manager.cc:1918] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:46:00.720409 6341 log.cc:1196] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/eef82b0e3b08400ab2997771964d39ca
I20260923 14:46:00.720701 6341 ts_tablet_manager.cc:1939] T eef82b0e3b08400ab2997771964d39ca P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting consensus metadata
I20260923 14:46:00.721920 5793 catalog_manager.cc:4841] TS 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195:41809): tablet eef82b0e3b08400ab2997771964d39ca (table test-table [id=e8d12a8289ce4dcda79031274b63e4f6]) successfully deleted
I20260923 14:46:00.835594 6323 raft_consensus.cc:482] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:46:00.835748 6323 raft_consensus.cc:504] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.836966 6323 leader_election.cc:288] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403), 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001)
I20260923 14:46:00.841671 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.837519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:58349 (local address 127.30.2.193:35403)
0923 14:46:00.837697 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.837702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.837720 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:46:00.837782 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.837786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.837834 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.837917 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:00.838311 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.839089 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.840529 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.841074 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.841153 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.841195 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.841474 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.841547 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 14:46:00.841693 6347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.837213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:35403 (local address 127.30.2.195:58349)
0923 14:46:00.837564 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:00.837581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:00.837698 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:00.838006 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:00.838009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:00.838029 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:00.838169 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.838179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.839228 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.839232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.840390 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.840400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.841272 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.841274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.841348 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.841365 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:46:00.841455 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 14:46:00.841526 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":60,"thread_start_us":84,"threads_started":1}
I20260923 14:46:00.841957 6313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.837885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:54479 (local address 127.30.2.194:44001)
0923 14:46:00.838014 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.838019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.838031 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:46:00.838080 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.838083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.838145 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.838227 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:00.838550 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.839227 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.840721 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.841299 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.841385 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.841433 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.841768 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.841837 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:46:00.842309 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.837260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:44001 (local address 127.30.2.195:54479)
0923 14:46:00.837746 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:00.837760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:00.837831 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:00.838266 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:00.838270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:00.838291 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:00.838412 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.838419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.839345 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.839349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.840555 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.840583 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.841536 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.841539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.841644 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.841664 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:46:00.842088 (+ 424us) client_negotiation.cc:240] Negotiation successful
0923 14:46:00.842177 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":117,"threads_started":1}
I20260923 14:46:00.842496 5963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc404b023fa4edbbbde3b87db19fdc2" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" is_pre_election: true
I20260923 14:46:00.842689 5963 raft_consensus.cc:2454] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 0.
I20260923 14:46:00.842841 6103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc404b023fa4edbbbde3b87db19fdc2" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771221bf1acd40d5a569a5b01cc76989" is_pre_election: true
I20260923 14:46:00.843096 6103 raft_consensus.cc:2454] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 0.
I20260923 14:46:00.843082 6177 leader_election.cc:302] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2057c7e00aa74c0f855e7dbf4e8db42a, b3cc10078fed4fdb901dccca9ee53a1d; no voters:
I20260923 14:46:00.843357 6323 raft_consensus.cc:2781] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:46:00.843410 6323 raft_consensus.cc:482] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:46:00.843456 6323 raft_consensus.cc:3037] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:00.844476 6323 raft_consensus.cc:504] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.844864 6323 leader_election.cc:288] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 election: Requested vote from peers b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403), 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001)
I20260923 14:46:00.845191 5963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc404b023fa4edbbbde3b87db19fdc2" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cc10078fed4fdb901dccca9ee53a1d"
I20260923 14:46:00.845247 6103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc404b023fa4edbbbde3b87db19fdc2" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771221bf1acd40d5a569a5b01cc76989"
I20260923 14:46:00.845346 5963 raft_consensus.cc:3037] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:00.845361 6103 raft_consensus.cc:3037] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:00.846308 5963 raft_consensus.cc:2454] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 1.
I20260923 14:46:00.846607 6177 leader_election.cc:302] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2057c7e00aa74c0f855e7dbf4e8db42a, b3cc10078fed4fdb901dccca9ee53a1d; no voters:
I20260923 14:46:00.846815 6323 raft_consensus.cc:2781] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:46:00.846997 6103 raft_consensus.cc:2454] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 1.
I20260923 14:46:00.847040 6323 raft_consensus.cc:686] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 LEADER]: Becoming Leader. State: Replica: 2057c7e00aa74c0f855e7dbf4e8db42a, State: Running, Role: LEADER
I20260923 14:46:00.847314 6323 consensus_queue.cc:237] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.850010 5804 catalog_manager.cc:5495] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195). New cstate: current_term: 1 leader_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } health_report { overall_health: HEALTHY } } }
Sep 23 14:46:00 dist-test-slave-wjf2 krb5kdc[5761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174759, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.254@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:46:00.906594 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.897805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40839 (local address 127.0.0.1:52436)
0923 14:46:00.898186 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:00.898205 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:00.898340 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:00.898633 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:00.898636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:00.899016 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:00.899114 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.899120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.900025 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.900028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.900605 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:00.900613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.901124 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.901126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:00.901208 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.901842 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:00.901854 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:00.903240 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:00.905096 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:00.905104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:00.905109 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:00.905333 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:00.905586 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:00.905588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:00.905594 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:00.905675 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:00.906152 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:00.906157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:00.906257 (+ 100us) client_negotiation.cc:769] Sending connection context
0923 14:46:00.906350 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 14:46:00.906458 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260923 14:46:00.906776 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:00.897866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.30.2.254:40839)
0923 14:46:00.898345 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:00.898353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:00.898371 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:46:00.898457 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:00.898460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:00.898513 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:00.898584 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:00.899289 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.899925 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.900758 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:00.901010 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:00.901103 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:00.903376 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:00.903379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:00.903381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:00.903400 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:00.904997 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:00.905425 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:00.905428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:00.905429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:00.905503 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:00.905799 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:00.905803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:00.905805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:00.906020 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:00.906139 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:00.906441 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 14:46:00.906545 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":87,"threads_started":1}
I20260923 14:46:00.910552 5804 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:52436:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 14:46:00.911289 5804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:46:00.916231 5943 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeee735ebf54dafba5cd0597e4fa3a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5aa37c242549438f810d82bb0d42d9d2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:46:00.916291 6083 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeee735ebf54dafba5cd0597e4fa3a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5aa37c242549438f810d82bb0d42d9d2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:46:00.916416 6223 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeee735ebf54dafba5cd0597e4fa3a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5aa37c242549438f810d82bb0d42d9d2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:46:00.916653 5943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeee735ebf54dafba5cd0597e4fa3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:00.916685 6083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeee735ebf54dafba5cd0597e4fa3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:00.916795 6223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeee735ebf54dafba5cd0597e4fa3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:00.919709 6317 tablet_bootstrap.cc:492] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989: Bootstrap starting.
I20260923 14:46:00.919894 6318 tablet_bootstrap.cc:492] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: Bootstrap starting.
I20260923 14:46:00.920627 6317 tablet_bootstrap.cc:654] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:00.920629 6319 tablet_bootstrap.cc:492] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a: Bootstrap starting.
I20260923 14:46:00.921118 6318 tablet_bootstrap.cc:654] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:00.921649 6319 tablet_bootstrap.cc:654] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:00.921959 6317 tablet_bootstrap.cc:492] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989: No bootstrap required, opened a new log
I20260923 14:46:00.922039 6317 ts_tablet_manager.cc:1397] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:46:00.922466 6318 tablet_bootstrap.cc:492] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: No bootstrap required, opened a new log
I20260923 14:46:00.922461 6317 raft_consensus.cc:348] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.922549 6318 ts_tablet_manager.cc:1397] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 14:46:00.922585 6317 raft_consensus.cc:374] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:00.922626 6317 raft_consensus.cc:729] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771221bf1acd40d5a569a5b01cc76989, State: Initialized, Role: FOLLOWER
I20260923 14:46:00.922751 6317 consensus_queue.cc:260] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.923012 6317 ts_tablet_manager.cc:1428] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:46:00.922979 6318 raft_consensus.cc:348] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.923079 6318 raft_consensus.cc:374] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:00.923122 6318 raft_consensus.cc:729] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cc10078fed4fdb901dccca9ee53a1d, State: Initialized, Role: FOLLOWER
I20260923 14:46:00.923173 6319 tablet_bootstrap.cc:492] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a: No bootstrap required, opened a new log
I20260923 14:46:00.923262 6319 ts_tablet_manager.cc:1397] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 14:46:00.923257 6318 consensus_queue.cc:260] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.923599 6324 tablet_replica.cc:439] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } }
I20260923 14:46:00.923622 6325 tablet_replica.cc:439] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } }
I20260923 14:46:00.923733 6325 tablet_replica.cc:442] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:46:00.923732 6324 tablet_replica.cc:442] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:46:00.923748 6319 raft_consensus.cc:348] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.923884 6319 raft_consensus.cc:374] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:00.923928 6319 raft_consensus.cc:729] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2057c7e00aa74c0f855e7dbf4e8db42a, State: Initialized, Role: FOLLOWER
I20260923 14:46:00.924072 6319 consensus_queue.cc:260] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:00.924372 6319 ts_tablet_manager.cc:1428] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:46:00.924311 6323 tablet_replica.cc:439] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } }
I20260923 14:46:00.924475 6323 tablet_replica.cc:442] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:46:00.925192 6318 ts_tablet_manager.cc:1428] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260923 14:46:01.208388 6325 raft_consensus.cc:482] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:46:01.208519 6325 raft_consensus.cc:504] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.209555 6325 leader_election.cc:288] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001), 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195:41809)
I20260923 14:46:01.215328 6313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.209944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:55447 (local address 127.30.2.194:44001)
0923 14:46:01.210098 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:01.210103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:01.210373 (+ 270us) server_negotiation.cc:407] Connection header received
0923 14:46:01.210507 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:01.210510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:01.210561 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:01.210649 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:01.211168 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.212002 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.213900 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.214522 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.214669 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.214712 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:01.215085 (+ 373us) server_negotiation.cc:299] Negotiation successful
0923 14:46:01.215190 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:46:01.215337 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.209931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:42755 (local address 127.30.2.195:41809)
0923 14:46:01.210237 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:01.210241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:01.210311 (+ 70us) server_negotiation.cc:407] Connection header received
0923 14:46:01.210482 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:01.210485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:01.210537 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:01.210641 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:01.211207 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.212001 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.213900 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.214529 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.214656 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.214712 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:01.215085 (+ 373us) server_negotiation.cc:299] Negotiation successful
0923 14:46:01.215211 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":82,"threads_started":1}
I20260923 14:46:01.215360 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.209825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:44001 (local address 127.30.2.193:55447)
0923 14:46:01.210312 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:01.210324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:01.210402 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:01.210673 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:01.210676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:01.210719 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:01.211033 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:01.211041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.212134 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.212139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:01.213745 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:01.213759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.214658 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.214662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:01.214892 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.214941 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 14:46:01.215021 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 14:46:01.215136 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":518,"spinlock_wait_cycles":7040,"thread_start_us":70,"threads_started":1}
I20260923 14:46:01.215395 6360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.209825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:41809 (local address 127.30.2.193:42755)
0923 14:46:01.210221 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:01.210238 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:01.210360 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:01.210673 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:01.210676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:01.210718 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:01.211033 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:01.211041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.212175 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.212178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:01.213745 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:01.213758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.214658 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.214662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:01.214878 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.214936 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 14:46:01.215021 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 14:46:01.215136 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":343,"spinlock_wait_cycles":31744,"thread_start_us":67,"threads_started":1}
I20260923 14:46:01.215926 6103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeee735ebf54dafba5cd0597e4fa3a3" candidate_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771221bf1acd40d5a569a5b01cc76989" is_pre_election: true
I20260923 14:46:01.216154 6103 raft_consensus.cc:2454] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3cc10078fed4fdb901dccca9ee53a1d in term 0.
I20260923 14:46:01.216359 6242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeee735ebf54dafba5cd0597e4fa3a3" candidate_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" is_pre_election: true
I20260923 14:46:01.216547 6242 raft_consensus.cc:2454] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3cc10078fed4fdb901dccca9ee53a1d in term 0.
I20260923 14:46:01.216575 5894 leader_election.cc:302] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771221bf1acd40d5a569a5b01cc76989, b3cc10078fed4fdb901dccca9ee53a1d; no voters:
I20260923 14:46:01.216902 6325 raft_consensus.cc:2781] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:46:01.217015 6325 raft_consensus.cc:482] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:46:01.217083 6325 raft_consensus.cc:3037] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:01.218304 6325 raft_consensus.cc:504] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.218729 6325 leader_election.cc:288] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [CANDIDATE]: Term 1 election: Requested vote from peers 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001), 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195:41809)
I20260923 14:46:01.219219 6103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeee735ebf54dafba5cd0597e4fa3a3" candidate_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771221bf1acd40d5a569a5b01cc76989"
I20260923 14:46:01.219269 6242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeee735ebf54dafba5cd0597e4fa3a3" candidate_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a"
I20260923 14:46:01.219339 6103 raft_consensus.cc:3037] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:01.219415 6242 raft_consensus.cc:3037] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:01.220517 6242 raft_consensus.cc:2454] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3cc10078fed4fdb901dccca9ee53a1d in term 1.
I20260923 14:46:01.220583 6103 raft_consensus.cc:2454] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3cc10078fed4fdb901dccca9ee53a1d in term 1.
I20260923 14:46:01.220921 5897 leader_election.cc:302] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2057c7e00aa74c0f855e7dbf4e8db42a, b3cc10078fed4fdb901dccca9ee53a1d; no voters:
I20260923 14:46:01.221190 6325 raft_consensus.cc:2781] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:46:01.221419 6325 raft_consensus.cc:686] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 LEADER]: Becoming Leader. State: Replica: b3cc10078fed4fdb901dccca9ee53a1d, State: Running, Role: LEADER
I20260923 14:46:01.221726 6325 consensus_queue.cc:237] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.223062 6363 tablet_replica.cc:439] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: TxnStatusTablet state changed. Reason: New leader b3cc10078fed4fdb901dccca9ee53a1d. Latest consensus state: current_term: 1 leader_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } }
I20260923 14:46:01.223194 6363 tablet_replica.cc:442] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:46:01.223661 6365 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 14:46:01.223869 6365 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 14:46:01.224500 5806 catalog_manager.cc:5495] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193). New cstate: current_term: 1 leader_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } health_report { overall_health: UNKNOWN } } }
I20260923 14:46:01.288915 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.284687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.30.2.193:35403)
0923 14:46:01.284857 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:01.284864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:01.284886 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:46:01.285090 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:01.285093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:01.285154 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:01.285290 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:46:01.285558 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.286217 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.287453 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.287677 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.287764 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.288122 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:46:01.288545 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:46:01.288646 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:01.288747 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 14:46:01.288798 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:46:01.288968 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.284550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:35403 (local address 127.0.0.1:42184)
0923 14:46:01.284764 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:01.284780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:01.284913 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:01.285310 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:01.285313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:01.285323 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:46:01.285436 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:01.285464 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.286327 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.286331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:01.287322 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:01.287333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.287806 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.287810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:01.287908 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.287967 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:46:01.288647 (+ 680us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:46:01.288664 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:46:01.288738 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 14:46:01.288825 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260923 14:46:01.292371 6242 raft_consensus.cc:1261] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Refusing update from remote peer b3cc10078fed4fdb901dccca9ee53a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:46:01.292557 6103 raft_consensus.cc:1261] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Refusing update from remote peer b3cc10078fed4fdb901dccca9ee53a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:46:01.292896 6363 consensus_queue.cc:1035] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:01.293484 6363 consensus_queue.cc:1035] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:01.294399 6323 tablet_replica.cc:439] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a: TxnStatusTablet state changed. Reason: New leader b3cc10078fed4fdb901dccca9ee53a1d. Latest consensus state: current_term: 1 leader_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } }
I20260923 14:46:01.294513 6323 tablet_replica.cc:442] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:46:01.296221 6363 tablet_replica.cc:439] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } }
I20260923 14:46:01.296329 6363 tablet_replica.cc:442] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:46:01.297106 6323 tablet_replica.cc:439] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } }
I20260923 14:46:01.297205 6323 tablet_replica.cc:442] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 2057c7e00aa74c0f855e7dbf4e8db42a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:46:01.297626 6324 tablet_replica.cc:439] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989: TxnStatusTablet state changed. Reason: New leader b3cc10078fed4fdb901dccca9ee53a1d. Latest consensus state: current_term: 1 leader_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } }
I20260923 14:46:01.297775 6324 tablet_replica.cc:442] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:46:01.302120 6325 tablet_replica.cc:439] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } }
I20260923 14:46:01.302232 6325 tablet_replica.cc:442] T 9aeee735ebf54dafba5cd0597e4fa3a3 P b3cc10078fed4fdb901dccca9ee53a1d: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:46:01.303753 6324 tablet_replica.cc:439] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } }
I20260923 14:46:01.303890 6324 tablet_replica.cc:442] T 9aeee735ebf54dafba5cd0597e4fa3a3 P 771221bf1acd40d5a569a5b01cc76989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:46:01.304302 6323 consensus_queue.cc:1035] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:01.308545 6348 consensus_queue.cc:1035] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:01.310179 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.302598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.30.2.195:41809)
0923 14:46:01.302725 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:01.302730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:01.305196 (+ 2466us) server_negotiation.cc:407] Connection header received
0923 14:46:01.305457 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:01.305462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:01.305534 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:01.305615 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:46:01.306063 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.306822 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.308100 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.308404 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.308506 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.309069 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:46:01.309659 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:46:01.309766 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:01.309961 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 14:46:01.310032 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:46:01.322113 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.317772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:35403 (local address 127.30.2.195:58603)
0923 14:46:01.318108 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:01.318124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:01.318243 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:01.318491 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:01.318494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:01.318502 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:46:01.319041 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:01.319052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.319747 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.319750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:01.320765 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:01.320776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.321252 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.321258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:01.321357 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.321398 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:46:01.321804 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:46:01.321813 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 14:46:01.321898 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 14:46:01.321977 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":117,"threads_started":1}
I20260923 14:46:01.322099 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.317901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:58603 (local address 127.30.2.193:35403)
0923 14:46:01.318029 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:01.318032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:01.318195 (+ 163us) server_negotiation.cc:407] Connection header received
0923 14:46:01.318351 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:01.318353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:01.318394 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:01.318458 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:46:01.319161 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.319636 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.320877 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.321085 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.321211 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.321555 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:46:01.321705 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:46:01.321800 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:01.321914 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 14:46:01.321960 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:46:01.348345 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.344250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:38735 (local address 127.30.2.195:41809)
0923 14:46:01.344406 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:01.344414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:01.344661 (+ 247us) server_negotiation.cc:407] Connection header received
0923 14:46:01.344816 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:01.344820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:01.344878 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:01.344955 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:46:01.345342 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.345991 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.347119 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.347354 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.347475 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.347848 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:46:01.347932 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:46:01.348017 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:01.348130 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 14:46:01.348180 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:46:01.348407 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.344169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:41809 (local address 127.30.2.193:38735)
0923 14:46:01.344570 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:01.344584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:01.344679 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:01.345091 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:01.345093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:01.345100 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:46:01.345208 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:01.345213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.346133 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.346137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:01.346991 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:01.347002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.347474 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.347479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:01.347593 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.347641 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:46:01.348039 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:46:01.348047 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 14:46:01.348120 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 14:46:01.348200 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":118,"threads_started":1}
I20260923 14:46:01.379251 5804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:46:01.379438 5804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:46:01.381969 5804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f: Sending DeleteTablet for 3 replicas of tablet bdc404b023fa4edbbbde3b87db19fdc2
I20260923 14:46:01.382753 6223 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc404b023fa4edbbbde3b87db19fdc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:01 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:45026
I20260923 14:46:01.382781 6083 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc404b023fa4edbbbde3b87db19fdc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:01 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42530
I20260923 14:46:01.383169 5943 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc404b023fa4edbbbde3b87db19fdc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:01 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42180
I20260923 14:46:01.383556 6388 tablet_replica.cc:330] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a: stopping tablet replica
I20260923 14:46:01.383759 6388 raft_consensus.cc:2229] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:46:01.384130 6388 raft_consensus.cc:2258] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:01.384967 6388 ts_tablet_manager.cc:1905] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:01.387431 6390 tablet_replica.cc:330] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d: stopping tablet replica
I20260923 14:46:01.387569 6390 raft_consensus.cc:2229] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:46:01.387703 6390 raft_consensus.cc:2258] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:01.388393 6388 ts_tablet_manager.cc:1918] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:46:01.388403 6390 ts_tablet_manager.cc:1905] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:01.388545 6388 log.cc:1196] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/bdc404b023fa4edbbbde3b87db19fdc2
I20260923 14:46:01.388919 6388 ts_tablet_manager.cc:1939] T bdc404b023fa4edbbbde3b87db19fdc2 P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting consensus metadata
I20260923 14:46:01.389802 5793 catalog_manager.cc:4841] TS 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195:41809): tablet bdc404b023fa4edbbbde3b87db19fdc2 (table test-table [id=00152946fa5547a0a0db38ce13553a40]) successfully deleted
I20260923 14:46:01.391080 6390 ts_tablet_manager.cc:1918] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:46:01.391160 6390 log.cc:1196] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/bdc404b023fa4edbbbde3b87db19fdc2
I20260923 14:46:01.391441 6390 ts_tablet_manager.cc:1939] T bdc404b023fa4edbbbde3b87db19fdc2 P b3cc10078fed4fdb901dccca9ee53a1d: Deleting consensus metadata
I20260923 14:46:01.391957 6389 tablet_replica.cc:330] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989: stopping tablet replica
I20260923 14:46:01.392107 6389 raft_consensus.cc:2229] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:46:01.392223 6389 raft_consensus.cc:2258] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:01.392972 6389 ts_tablet_manager.cc:1905] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:01.396607 6389 ts_tablet_manager.cc:1918] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:46:01.396694 6389 log.cc:1196] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/bdc404b023fa4edbbbde3b87db19fdc2
I20260923 14:46:01.396672 5793 catalog_manager.cc:4841] TS b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403): tablet bdc404b023fa4edbbbde3b87db19fdc2 (table test-table [id=00152946fa5547a0a0db38ce13553a40]) successfully deleted
I20260923 14:46:01.396965 6389 ts_tablet_manager.cc:1939] T bdc404b023fa4edbbbde3b87db19fdc2 P 771221bf1acd40d5a569a5b01cc76989: Deleting consensus metadata
I20260923 14:46:01.397595 5790 catalog_manager.cc:4841] TS 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001): tablet bdc404b023fa4edbbbde3b87db19fdc2 (table test-table [id=00152946fa5547a0a0db38ce13553a40]) successfully deleted
Sep 23 14:46:01 dist-test-slave-wjf2 krb5kdc[5761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 14:46:01 dist-test-slave-wjf2 krb5kdc[5761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:01 dist-test-slave-wjf2 krb5kdc[5761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 14:46:01.427462 6398 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:01.418927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.30.2.254:40839)
0923 14:46:01.419195 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:01.419201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:01.419314 (+ 113us) server_negotiation.cc:407] Connection header received
0923 14:46:01.419508 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:01.419511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:01.419582 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:01.419679 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:01.420467 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.421102 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.422268 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.422509 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.422626 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.427292 (+ 4666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52444: BlockingRecv error: recv got EOF from 127.0.0.1:52444 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
Sep 23 14:46:01 dist-test-slave-wjf2 krb5kdc[5761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM
I20260923 14:46:01.438294 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.429319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.30.2.254:40839)
0923 14:46:01.429457 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:01.429462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:01.429625 (+ 163us) server_negotiation.cc:407] Connection header received
0923 14:46:01.429728 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:01.429730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:01.429773 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:01.429840 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:01.430556 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.431063 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.432197 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.432435 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.432516 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.435012 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:01.435016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:01.435018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:01.435039 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:01.436626 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:01.437057 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:01.437061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:01.437063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:01.437148 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:01.437437 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:01.437440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:01.437442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:01.437626 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:01.437767 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:01.438007 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 14:46:01.438129 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:46:01.440927 5804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:46:01.441442 5804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:46:01.446257 5943 tablet_service.cc:1467] Processing CreateTablet for tablet 90b3ff1f438f4c448d0720299796a429 (DEFAULT_TABLE table=test-table [id=cd04a9a1a8c84b5c8cf76802d0e11457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:01.446410 6083 tablet_service.cc:1467] Processing CreateTablet for tablet 90b3ff1f438f4c448d0720299796a429 (DEFAULT_TABLE table=test-table [id=cd04a9a1a8c84b5c8cf76802d0e11457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:01.446678 5943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b3ff1f438f4c448d0720299796a429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:01.446708 6083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b3ff1f438f4c448d0720299796a429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:01.446707 6223 tablet_service.cc:1467] Processing CreateTablet for tablet 90b3ff1f438f4c448d0720299796a429 (DEFAULT_TABLE table=test-table [id=cd04a9a1a8c84b5c8cf76802d0e11457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:01.446981 6223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b3ff1f438f4c448d0720299796a429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:01.449509 6404 tablet_bootstrap.cc:492] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989: Bootstrap starting.
I20260923 14:46:01.450156 6406 tablet_bootstrap.cc:492] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a: Bootstrap starting.
I20260923 14:46:01.450985 6404 tablet_bootstrap.cc:654] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:01.451202 6406 tablet_bootstrap.cc:654] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:01.452059 6405 tablet_bootstrap.cc:492] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d: Bootstrap starting.
I20260923 14:46:01.452389 6404 tablet_bootstrap.cc:492] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989: No bootstrap required, opened a new log
I20260923 14:46:01.452476 6404 ts_tablet_manager.cc:1397] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 14:46:01.453023 6405 tablet_bootstrap.cc:654] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:01.453125 6404 raft_consensus.cc:348] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.453240 6404 raft_consensus.cc:374] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:01.453272 6404 raft_consensus.cc:729] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771221bf1acd40d5a569a5b01cc76989, State: Initialized, Role: FOLLOWER
I20260923 14:46:01.453423 6404 consensus_queue.cc:260] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.453694 6404 ts_tablet_manager.cc:1428] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:46:01.453998 6406 tablet_bootstrap.cc:492] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a: No bootstrap required, opened a new log
I20260923 14:46:01.454083 6406 ts_tablet_manager.cc:1397] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 14:46:01.454527 6406 raft_consensus.cc:348] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.454651 6406 raft_consensus.cc:374] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:01.454694 6406 raft_consensus.cc:729] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2057c7e00aa74c0f855e7dbf4e8db42a, State: Initialized, Role: FOLLOWER
I20260923 14:46:01.454836 6406 consensus_queue.cc:260] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.455188 6406 ts_tablet_manager.cc:1428] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:46:01.455678 6405 tablet_bootstrap.cc:492] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d: No bootstrap required, opened a new log
I20260923 14:46:01.455771 6405 ts_tablet_manager.cc:1397] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 14:46:01.457165 6405 raft_consensus.cc:348] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.457289 6405 raft_consensus.cc:374] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:01.457335 6405 raft_consensus.cc:729] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cc10078fed4fdb901dccca9ee53a1d, State: Initialized, Role: FOLLOWER
I20260923 14:46:01.457472 6405 consensus_queue.cc:260] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.457782 6405 ts_tablet_manager.cc:1428] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 14:46:01.477285 6348 raft_consensus.cc:482] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:46:01.477473 6348 raft_consensus.cc:504] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.478004 6348 leader_election.cc:288] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403), 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001)
I20260923 14:46:01.478416 5963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b3ff1f438f4c448d0720299796a429" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" is_pre_election: true
I20260923 14:46:01.478482 6102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b3ff1f438f4c448d0720299796a429" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771221bf1acd40d5a569a5b01cc76989" is_pre_election: true
I20260923 14:46:01.478593 5963 raft_consensus.cc:2454] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 0.
I20260923 14:46:01.478646 6102 raft_consensus.cc:2454] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 0.
I20260923 14:46:01.478904 6174 leader_election.cc:302] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2057c7e00aa74c0f855e7dbf4e8db42a, 771221bf1acd40d5a569a5b01cc76989; no voters:
I20260923 14:46:01.479100 6348 raft_consensus.cc:2781] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:46:01.479173 6348 raft_consensus.cc:482] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:46:01.479228 6348 raft_consensus.cc:3037] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:01.480288 6348 raft_consensus.cc:504] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.480665 6348 leader_election.cc:288] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 election: Requested vote from peers b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403), 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001)
I20260923 14:46:01.481030 5963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b3ff1f438f4c448d0720299796a429" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cc10078fed4fdb901dccca9ee53a1d"
I20260923 14:46:01.481163 5963 raft_consensus.cc:3037] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:01.481197 6102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b3ff1f438f4c448d0720299796a429" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771221bf1acd40d5a569a5b01cc76989"
I20260923 14:46:01.481314 6102 raft_consensus.cc:3037] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:01.482049 5963 raft_consensus.cc:2454] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 1.
I20260923 14:46:01.482429 6177 leader_election.cc:302] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2057c7e00aa74c0f855e7dbf4e8db42a, b3cc10078fed4fdb901dccca9ee53a1d; no voters:
I20260923 14:46:01.482472 6102 raft_consensus.cc:2454] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 1.
I20260923 14:46:01.482789 6348 raft_consensus.cc:2781] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:46:01.482897 6348 raft_consensus.cc:686] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 LEADER]: Becoming Leader. State: Replica: 2057c7e00aa74c0f855e7dbf4e8db42a, State: Running, Role: LEADER
I20260923 14:46:01.483042 6348 consensus_queue.cc:237] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } }
I20260923 14:46:01.484684 5806 catalog_manager.cc:5495] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195). New cstate: current_term: 1 leader_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } health_report { overall_health: HEALTHY } } }
I20260923 14:46:01.497623 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:01.494140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.30.2.195:41809)
0923 14:46:01.494262 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:01.494266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:01.494278 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:46:01.494415 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:01.494417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:01.494457 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:01.494516 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:46:01.494806 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.495294 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.496580 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:01.496784 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:01.496861 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:01.497164 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:46:01.497230 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:46:01.497278 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:01.497409 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 14:46:01.497450 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:46:01.499413 6102 raft_consensus.cc:1261] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Refusing update from remote peer 2057c7e00aa74c0f855e7dbf4e8db42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:46:01.499428 5963 raft_consensus.cc:1261] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Refusing update from remote peer 2057c7e00aa74c0f855e7dbf4e8db42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:46:01.499867 6348 consensus_queue.cc:1035] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:01.500022 6383 consensus_queue.cc:1035] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:01.505959 5804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:46:01.506138 5804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:46:01.509404 5804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f: Sending DeleteTablet for 3 replicas of tablet 90b3ff1f438f4c448d0720299796a429
I20260923 14:46:01.510090 5943 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b3ff1f438f4c448d0720299796a429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:01 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42180
I20260923 14:46:01.510267 6083 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b3ff1f438f4c448d0720299796a429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:01 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42530
I20260923 14:46:01.510394 6390 tablet_replica.cc:330] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d: stopping tablet replica
I20260923 14:46:01.510519 6390 raft_consensus.cc:2229] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:46:01.510600 6223 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b3ff1f438f4c448d0720299796a429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:01 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:45026
I20260923 14:46:01.510658 6390 raft_consensus.cc:2258] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:01.510769 6388 tablet_replica.cc:330] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a: stopping tablet replica
I20260923 14:46:01.510821 6389 tablet_replica.cc:330] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989: stopping tablet replica
I20260923 14:46:01.510875 6388 raft_consensus.cc:2229] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:46:01.510926 6389 raft_consensus.cc:2229] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:46:01.511041 6389 raft_consensus.cc:2258] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:01.511040 6388 raft_consensus.cc:2258] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:01.511749 6388 ts_tablet_manager.cc:1905] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:01.512156 6389 ts_tablet_manager.cc:1905] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:01.512804 5801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:46:01.513222 5801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:46:01.514426 6388 ts_tablet_manager.cc:1918] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:46:01.514678 6389 ts_tablet_manager.cc:1918] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:46:01.514760 6389 log.cc:1196] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/90b3ff1f438f4c448d0720299796a429
I20260923 14:46:01.515363 6389 ts_tablet_manager.cc:1939] T 90b3ff1f438f4c448d0720299796a429 P 771221bf1acd40d5a569a5b01cc76989: Deleting consensus metadata
I20260923 14:46:01.516050 5790 catalog_manager.cc:4841] TS 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001): tablet 90b3ff1f438f4c448d0720299796a429 (table test-table [id=cd04a9a1a8c84b5c8cf76802d0e11457]) successfully deleted
I20260923 14:46:01.514707 6388 log.cc:1196] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/90b3ff1f438f4c448d0720299796a429
I20260923 14:46:01.516963 6388 ts_tablet_manager.cc:1939] T 90b3ff1f438f4c448d0720299796a429 P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting consensus metadata
I20260923 14:46:01.517633 5793 catalog_manager.cc:4841] TS 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195:41809): tablet 90b3ff1f438f4c448d0720299796a429 (table test-table [id=cd04a9a1a8c84b5c8cf76802d0e11457]) successfully deleted
I20260923 14:46:01.518970 6390 ts_tablet_manager.cc:1905] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:01.520506 6223 tablet_service.cc:1467] Processing CreateTablet for tablet 13dc4d68c13f40c68f86e6b0652a9dc6 (DEFAULT_TABLE table=test-table [id=5a1ca2b3579e4a25af8c361ac38a78b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:01.520850 6223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13dc4d68c13f40c68f86e6b0652a9dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:01.521659 6390 ts_tablet_manager.cc:1918] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:46:01.521751 6390 log.cc:1196] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/90b3ff1f438f4c448d0720299796a429
I20260923 14:46:01.522022 6390 ts_tablet_manager.cc:1939] T 90b3ff1f438f4c448d0720299796a429 P b3cc10078fed4fdb901dccca9ee53a1d: Deleting consensus metadata
I20260923 14:46:01.522758 5793 catalog_manager.cc:4841] TS b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403): tablet 90b3ff1f438f4c448d0720299796a429 (table test-table [id=cd04a9a1a8c84b5c8cf76802d0e11457]) successfully deleted
I20260923 14:46:01.523010 6406 tablet_bootstrap.cc:492] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a: Bootstrap starting.
I20260923 14:46:01.523608 5943 tablet_service.cc:1467] Processing CreateTablet for tablet 13dc4d68c13f40c68f86e6b0652a9dc6 (DEFAULT_TABLE table=test-table [id=5a1ca2b3579e4a25af8c361ac38a78b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:01.523764 6083 tablet_service.cc:1467] Processing CreateTablet for tablet 13dc4d68c13f40c68f86e6b0652a9dc6 (DEFAULT_TABLE table=test-table [id=5a1ca2b3579e4a25af8c361ac38a78b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:01.523864 5943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13dc4d68c13f40c68f86e6b0652a9dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:01.524019 6083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13dc4d68c13f40c68f86e6b0652a9dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:01.524194 6406 tablet_bootstrap.cc:654] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:01.525426 6406 tablet_bootstrap.cc:492] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a: No bootstrap required, opened a new log
I20260923 14:46:01.525501 6406 ts_tablet_manager.cc:1397] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 14:46:01.525985 6406 raft_consensus.cc:348] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } }
I20260923 14:46:01.526094 6406 raft_consensus.cc:374] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:01.526130 6406 raft_consensus.cc:729] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2057c7e00aa74c0f855e7dbf4e8db42a, State: Initialized, Role: FOLLOWER
I20260923 14:46:01.526252 6406 consensus_queue.cc:260] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } }
I20260923 14:46:01.526377 6405 tablet_bootstrap.cc:492] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d: Bootstrap starting.
I20260923 14:46:01.526563 6406 ts_tablet_manager.cc:1428] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 14:46:01.527123 6405 tablet_bootstrap.cc:654] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:01.527663 6404 tablet_bootstrap.cc:492] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989: Bootstrap starting.
I20260923 14:46:01.528615 6404 tablet_bootstrap.cc:654] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:01.529853 6405 tablet_bootstrap.cc:492] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d: No bootstrap required, opened a new log
I20260923 14:46:01.529943 6405 ts_tablet_manager.cc:1397] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260923 14:46:01.530285 6405 raft_consensus.cc:348] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } }
I20260923 14:46:01.530396 6405 raft_consensus.cc:374] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:01.530435 6405 raft_consensus.cc:729] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cc10078fed4fdb901dccca9ee53a1d, State: Initialized, Role: FOLLOWER
I20260923 14:46:01.530586 6405 consensus_queue.cc:260] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } }
I20260923 14:46:01.530858 6405 ts_tablet_manager.cc:1428] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 14:46:01.531454 6404 tablet_bootstrap.cc:492] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989: No bootstrap required, opened a new log
I20260923 14:46:01.531543 6404 ts_tablet_manager.cc:1397] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260923 14:46:01.531936 6404 raft_consensus.cc:348] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } }
I20260923 14:46:01.532043 6404 raft_consensus.cc:374] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:01.532084 6404 raft_consensus.cc:729] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771221bf1acd40d5a569a5b01cc76989, State: Initialized, Role: FOLLOWER
I20260923 14:46:01.532215 6404 consensus_queue.cc:260] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } }
I20260923 14:46:01.532471 6404 ts_tablet_manager.cc:1428] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:46:01.600255 6383 raft_consensus.cc:482] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:46:01.600399 6383 raft_consensus.cc:504] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } }
I20260923 14:46:01.600857 6383 leader_election.cc:288] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403), 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001)
I20260923 14:46:01.601364 5963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13dc4d68c13f40c68f86e6b0652a9dc6" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" is_pre_election: true
I20260923 14:46:01.601364 6102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13dc4d68c13f40c68f86e6b0652a9dc6" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771221bf1acd40d5a569a5b01cc76989" is_pre_election: true
I20260923 14:46:01.601524 6102 raft_consensus.cc:2454] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 0.
I20260923 14:46:01.601524 5963 raft_consensus.cc:2454] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 0.
I20260923 14:46:01.601854 6177 leader_election.cc:302] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2057c7e00aa74c0f855e7dbf4e8db42a, b3cc10078fed4fdb901dccca9ee53a1d; no voters:
I20260923 14:46:01.602070 6383 raft_consensus.cc:2781] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:46:01.602138 6383 raft_consensus.cc:482] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:46:01.602185 6383 raft_consensus.cc:3037] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:01.603281 6383 raft_consensus.cc:504] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } }
I20260923 14:46:01.603636 6383 leader_election.cc:288] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 election: Requested vote from peers b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403), 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001)
I20260923 14:46:01.604045 6102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13dc4d68c13f40c68f86e6b0652a9dc6" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771221bf1acd40d5a569a5b01cc76989"
I20260923 14:46:01.604046 5963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13dc4d68c13f40c68f86e6b0652a9dc6" candidate_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cc10078fed4fdb901dccca9ee53a1d"
I20260923 14:46:01.604182 6102 raft_consensus.cc:3037] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:01.604194 5963 raft_consensus.cc:3037] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:01.605080 5963 raft_consensus.cc:2454] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 1.
I20260923 14:46:01.605259 6102 raft_consensus.cc:2454] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2057c7e00aa74c0f855e7dbf4e8db42a in term 1.
I20260923 14:46:01.605410 6177 leader_election.cc:302] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2057c7e00aa74c0f855e7dbf4e8db42a, b3cc10078fed4fdb901dccca9ee53a1d; no voters:
I20260923 14:46:01.605607 6383 raft_consensus.cc:2781] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:46:01.605716 6383 raft_consensus.cc:686] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 LEADER]: Becoming Leader. State: Replica: 2057c7e00aa74c0f855e7dbf4e8db42a, State: Running, Role: LEADER
I20260923 14:46:01.605868 6383 consensus_queue.cc:237] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } }
I20260923 14:46:01.607559 5801 catalog_manager.cc:5495] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195). New cstate: current_term: 1 leader_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2057c7e00aa74c0f855e7dbf4e8db42a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 } health_report { overall_health: UNKNOWN } } }
I20260923 14:46:01.639922 6102 raft_consensus.cc:1261] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Refusing update from remote peer 2057c7e00aa74c0f855e7dbf4e8db42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:46:01.639922 5963 raft_consensus.cc:1261] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Refusing update from remote peer 2057c7e00aa74c0f855e7dbf4e8db42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:46:01.640352 6383 consensus_queue.cc:1035] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3cc10078fed4fdb901dccca9ee53a1d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:01.640501 6348 consensus_queue.cc:1035] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "771221bf1acd40d5a569a5b01cc76989" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:01.669493 5804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:46:01.669658 5804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:46:01.671686 5804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 326417a82b8f4a629d88c9f665647d6f: Sending DeleteTablet for 3 replicas of tablet 13dc4d68c13f40c68f86e6b0652a9dc6
I20260923 14:46:01.672287 5943 tablet_service.cc:1514] Processing DeleteTablet for tablet 13dc4d68c13f40c68f86e6b0652a9dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:01 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42180
I20260923 14:46:01.672287 6223 tablet_service.cc:1514] Processing DeleteTablet for tablet 13dc4d68c13f40c68f86e6b0652a9dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:01 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:45026
I20260923 14:46:01.672500 6390 tablet_replica.cc:330] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d: stopping tablet replica
I20260923 14:46:01.672504 6083 tablet_service.cc:1514] Processing DeleteTablet for tablet 13dc4d68c13f40c68f86e6b0652a9dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:01 UTC) from {username='oryx', principal='oryx/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42530
I20260923 14:46:01.672631 6390 raft_consensus.cc:2229] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:46:01.672654 6389 tablet_replica.cc:330] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989: stopping tablet replica
I20260923 14:46:01.672736 6390 raft_consensus.cc:2258] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:01.672777 6389 raft_consensus.cc:2229] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:46:01.672899 6389 raft_consensus.cc:2258] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:01.673180 6388 tablet_replica.cc:330] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a: stopping tablet replica
I20260923 14:46:01.673308 6388 raft_consensus.cc:2229] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:46:01.673463 6390 ts_tablet_manager.cc:1905] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:01.673501 6388 raft_consensus.cc:2258] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:01.674258 6388 ts_tablet_manager.cc:1905] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:01.675994 6390 ts_tablet_manager.cc:1918] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:46:01.676061 6390 log.cc:1196] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/13dc4d68c13f40c68f86e6b0652a9dc6
I20260923 14:46:01.676280 6390 ts_tablet_manager.cc:1939] T 13dc4d68c13f40c68f86e6b0652a9dc6 P b3cc10078fed4fdb901dccca9ee53a1d: Deleting consensus metadata
I20260923 14:46:01.676930 5793 catalog_manager.cc:4841] TS b3cc10078fed4fdb901dccca9ee53a1d (127.30.2.193:35403): tablet 13dc4d68c13f40c68f86e6b0652a9dc6 (table test-table [id=5a1ca2b3579e4a25af8c361ac38a78b3]) successfully deleted
I20260923 14:46:01.677259 6388 ts_tablet_manager.cc:1918] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:46:01.677403 6388 log.cc:1196] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/13dc4d68c13f40c68f86e6b0652a9dc6
I20260923 14:46:01.677408 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 5878
I20260923 14:46:01.678644 6388 ts_tablet_manager.cc:1939] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 2057c7e00aa74c0f855e7dbf4e8db42a: Deleting consensus metadata
I20260923 14:46:01.678645 6389 ts_tablet_manager.cc:1905] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:01.679342 5793 catalog_manager.cc:4841] TS 2057c7e00aa74c0f855e7dbf4e8db42a (127.30.2.195:41809): tablet 13dc4d68c13f40c68f86e6b0652a9dc6 (table test-table [id=5a1ca2b3579e4a25af8c361ac38a78b3]) successfully deleted
I20260923 14:46:01.681242 6389 ts_tablet_manager.cc:1918] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:46:01.681308 6389 log.cc:1196] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/13dc4d68c13f40c68f86e6b0652a9dc6
I20260923 14:46:01.681542 6389 ts_tablet_manager.cc:1939] T 13dc4d68c13f40c68f86e6b0652a9dc6 P 771221bf1acd40d5a569a5b01cc76989: Deleting consensus metadata
I20260923 14:46:01.682154 5790 catalog_manager.cc:4841] TS 771221bf1acd40d5a569a5b01cc76989 (127.30.2.194:44001): tablet 13dc4d68c13f40c68f86e6b0652a9dc6 (table test-table [id=5a1ca2b3579e4a25af8c361ac38a78b3]) successfully deleted
I20260923 14:46:01.688678 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 6018
I20260923 14:46:01.695663 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 6158
I20260923 14:46:01.702538 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 5774
2026-09-23T14:46:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3689 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:46:01 dist-test-slave-wjf2 krb5kdc[6424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:46:01 dist-test-slave-wjf2 krb5kdc[6424](info): set up 2 sockets
Sep 23 14:46:01 dist-test-slave-wjf2 krb5kdc[6424](info): commencing operation
krb5kdc: starting...
W20260923 14:46:02.738374 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:46:02 dist-test-slave-wjf2 krb5kdc[6424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:46:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:46:02.863837 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44863
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:44863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:46:02.976244 6437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:02.976483 6437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:02.976521 6437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:02.980449 6437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:46:02.980499 6437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:02.980518 6437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:02.980535 6437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:46:02.980551 6437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:46:02.985545 6437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:44863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:02.986465 6437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:02.987425 6437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:02.995472 6443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:02.995561 6437 server_base.cc:1034] running on GCE node
W20260923 14:46:02.995460 6442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:02.995476 6445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:02.996284 6437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:02.997344 6437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:02.998553 6437 hybrid_clock.cc:648] HybridClock initialized: now 1790174762998541 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T14:46:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1308 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:46:03 dist-test-slave-wjf2 krb5kdc[6450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:46:03 dist-test-slave-wjf2 krb5kdc[6450](info): set up 2 sockets
Sep 23 14:46:03 dist-test-slave-wjf2 krb5kdc[6450](info): commencing operation
krb5kdc: starting...
W20260923 14:46:04.134011 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:46:04 dist-test-slave-wjf2 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:46:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:46:04.256834 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:41823
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:41823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:46:04.365731 6463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:04.365990 6463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:04.366030 6463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:04.370056 6463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:46:04.370112 6463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:04.370132 6463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:04.370193 6463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:46:04.370216 6463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:46:04.375402 6463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:41823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:04.376417 6463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:04.377421 6463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:04.386030 6469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:04.386029 6468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:04.386210 6471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:04.386379 6463 server_base.cc:1034] running on GCE node
I20260923 14:46:04.386965 6463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:04.388099 6463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:04.389318 6463 hybrid_clock.cc:648] HybridClock initialized: now 1790174764389302 us; error 25 us; skew 500 ppm
Sep 23 14:46:04 dist-test-slave-wjf2 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174764, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:04.392078 6463 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:46:04.392915 6463 webserver.cc:460] Webserver started at http://127.30.2.254:35057/ using document root <none> and password file <none>
I20260923 14:46:04.393476 6463 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:04.393579 6463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:04.393806 6463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:04.395869 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "04c4dcb835f64db1b37da0d4fc84ffcd"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "9b2f9bb2c266d2634ff29757e597e261"
server_key_iv: "d42131a2404c5474299e30430c2358db"
server_key_version: "encryptionkey@0"
I20260923 14:46:04.396430 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "04c4dcb835f64db1b37da0d4fc84ffcd"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "9b2f9bb2c266d2634ff29757e597e261"
server_key_iv: "d42131a2404c5474299e30430c2358db"
server_key_version: "encryptionkey@0"
I20260923 14:46:04.400091 6463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:46:04.402307 6478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:04.403403 6463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:46:04.403538 6463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "04c4dcb835f64db1b37da0d4fc84ffcd"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "9b2f9bb2c266d2634ff29757e597e261"
server_key_iv: "d42131a2404c5474299e30430c2358db"
server_key_version: "encryptionkey@0"
I20260923 14:46:04.403640 6463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:04.422859 6463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:04.425680 6463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:04.425890 6463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:04.432924 6463 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:41823
I20260923 14:46:04.432934 6530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:41823 every 8 connection(s)
I20260923 14:46:04.433903 6463 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:46:04.436714 6531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:04.442116 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd: Bootstrap starting.
I20260923 14:46:04.442708 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 6463
I20260923 14:46:04.442844 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:46:04.443148 30731 external_mini_cluster.cc:1428] Setting key b105b198e84cf84965d8bd7dcfbdc84b
I20260923 14:46:04.445001 6531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:04.445751 6531 log.cc:824] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd: Log is configured to *not* fsync() on all Append() calls
I20260923 14:46:04.447988 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd: No bootstrap required, opened a new log
Sep 23 14:46:04 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:46:04.452627 6531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } }
I20260923 14:46:04.452878 6531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:04.452939 6531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c4dcb835f64db1b37da0d4fc84ffcd, State: Initialized, Role: FOLLOWER
I20260923 14:46:04.453449 6531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } }
I20260923 14:46:04.453644 6531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:46:04.453720 6531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:46:04.453864 6531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:04.455013 6531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } }
I20260923 14:46:04.455368 6531 leader_election.cc:302] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04c4dcb835f64db1b37da0d4fc84ffcd; no voters:
I20260923 14:46:04.455674 6531 leader_election.cc:288] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:46:04.455819 6536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:46:04.456115 6536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 1 LEADER]: Becoming Leader. State: Replica: 04c4dcb835f64db1b37da0d4fc84ffcd, State: Running, Role: LEADER
I20260923 14:46:04.456477 6536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } }
I20260923 14:46:04.457005 6531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:46:04.458138 6536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c4dcb835f64db1b37da0d4fc84ffcd. Latest consensus state: current_term: 1 leader_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } } }
I20260923 14:46:04.458276 6536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [sys.catalog]: This master's current role is: LEADER
I20260923 14:46:04.458712 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } } }
I20260923 14:46:04.458889 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [sys.catalog]: This master's current role is: LEADER
I20260923 14:46:04.459144 6545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:46:04.459681 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:04.444510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.30.2.254:41823)
0923 14:46:04.444938 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:04.444962 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:04.444996 (+ 34us) server_negotiation.cc:407] Connection header received
0923 14:46:04.445657 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:04.445673 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:04.446055 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:04.446475 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:04.447347 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:04.448420 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:04.449584 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:04.450032 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:04.450148 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:04.452917 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:04.452926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:04.452942 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:04.452979 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:04.455042 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:04.455652 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:04.455661 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:04.455669 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:04.455880 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:04.456351 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:04.456356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:04.456358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:04.456985 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:04.457312 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:04.457648 (+ 336us) server_negotiation.cc:299] Negotiation successful
0923 14:46:04.458477 (+ 829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":147,"threads_started":1}
I20260923 14:46:04.463049 6545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:46:04.469688 6545 catalog_manager.cc:1347] Generated new cluster ID: 3c7a255c66a54c3590692e9e4c13b14e
I20260923 14:46:04.469781 6545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:46:04.490648 6545 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:46:04.491748 6545 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:46:04.496986 6545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd: Generated new TSK 0
I20260923 14:46:04.497777 6545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:46:04.527962 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:41823
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:46:04.637352 6557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:04.637585 6557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:04.637626 6557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:04.641682 6557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:04.641743 6557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:04.641799 6557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:46:04.647071 6557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:41823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:04.648130 6557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:04.649169 6557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:04.659531 6563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:04.659549 6562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:04.659543 6565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:04.659844 6557 server_base.cc:1034] running on GCE node
I20260923 14:46:04.660326 6557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:04.660985 6557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:04.662194 6557 hybrid_clock.cc:648] HybridClock initialized: now 1790174764662177 us; error 26 us; skew 500 ppm
Sep 23 14:46:04 dist-test-slave-wjf2 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174764, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:04.665126 6557 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:46:04.666026 6557 webserver.cc:460] Webserver started at http://127.30.2.193:39153/ using document root <none> and password file <none>
I20260923 14:46:04.666612 6557 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:04.666728 6557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:04.666982 6557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:04.669389 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "9d1ef168d26a46f4bb806c14862dc80b"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "7e1ebb78998bdc92d6a952af87d8db06"
server_key_iv: "87b2a2a803ba94c97f1d6c57e9438991"
server_key_version: "encryptionkey@0"
I20260923 14:46:04.670032 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "9d1ef168d26a46f4bb806c14862dc80b"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "7e1ebb78998bdc92d6a952af87d8db06"
server_key_iv: "87b2a2a803ba94c97f1d6c57e9438991"
server_key_version: "encryptionkey@0"
I20260923 14:46:04.673920 6557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:46:04.676486 6572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:04.677930 6557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:46:04.678061 6557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "9d1ef168d26a46f4bb806c14862dc80b"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "7e1ebb78998bdc92d6a952af87d8db06"
server_key_iv: "87b2a2a803ba94c97f1d6c57e9438991"
server_key_version: "encryptionkey@0"
I20260923 14:46:04.678176 6557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:04.699892 6557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:04.703284 6557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:04.703519 6557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:04.704208 6557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:04.705129 6557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:04.705205 6557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:04.705286 6557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:04.705341 6557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:04.715071 6557 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:33809
I20260923 14:46:04.715085 6685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:33809 every 8 connection(s)
I20260923 14:46:04.716185 6557 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:46:04.724180 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 6557
I20260923 14:46:04.724349 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:46:04.724730 30731 external_mini_cluster.cc:1428] Setting key 54349152b3a1f6b8fc837885adf2f12c
Sep 23 14:46:04 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174764, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:46:04.731194 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:04.718240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:47209 (local address 127.30.2.254:41823)
0923 14:46:04.718401 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:04.718405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:04.719241 (+ 836us) server_negotiation.cc:407] Connection header received
0923 14:46:04.720734 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:04.720738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:04.720794 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:04.720876 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:04.722034 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:04.722606 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:04.723717 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:04.724007 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:04.724116 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:04.727200 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:04.727203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:04.727205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:04.727228 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:04.728984 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:04.729517 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:04.729524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:04.729528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:04.729617 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:04.729954 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:04.729960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:04.729965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:04.730209 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:04.730313 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:04.730799 (+ 486us) server_negotiation.cc:299] Negotiation successful
0923 14:46:04.730948 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 14:46:04.731989 6688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:04.718611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:47209)
0923 14:46:04.719068 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:04.719112 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:04.720480 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:04.721059 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:04.721066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:04.721412 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:04.721885 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:04.721899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:04.722733 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:04.722738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:04.723590 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:04.723599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:04.724165 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:04.724170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:04.724367 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:04.725055 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:04.725072 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:04.727026 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:04.729136 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:04.729141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:04.729153 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:04.729411 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:04.729722 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:04.729724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:04.729726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:04.729857 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:04.730312 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:04.730317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:04.730545 (+ 228us) client_negotiation.cc:769] Sending connection context
0923 14:46:04.730712 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 14:46:04.731035 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260923 14:46:04.733286 6686 heartbeater.cc:347] Connected to a master server at 127.30.2.254:41823
I20260923 14:46:04.733567 6686 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:04.734066 6686 heartbeater.cc:511] Master 127.30.2.254:41823 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:46:04.735719 6495 ts_manager.cc:194] Registered new tserver with Master: 9d1ef168d26a46f4bb806c14862dc80b (127.30.2.193:33809)
I20260923 14:46:04.737066 6495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:47209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:46:04.754992 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:41823
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:46:04.867331 6691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:04.867560 6691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:04.867600 6691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:04.871537 6691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:04.871584 6691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:04.871639 6691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:46:04.876564 6691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:41823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:04.877501 6691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:04.878455 6691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:04.888075 6696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:04.888123 6697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:04.888082 6699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:04.888281 6691 server_base.cc:1034] running on GCE node
I20260923 14:46:04.888754 6691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:04.889344 6691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:04.890501 6691 hybrid_clock.cc:648] HybridClock initialized: now 1790174764890485 us; error 34 us; skew 500 ppm
Sep 23 14:46:04 dist-test-slave-wjf2 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174764, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:04.893198 6691 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:46:04.893963 6691 webserver.cc:460] Webserver started at http://127.30.2.194:41415/ using document root <none> and password file <none>
I20260923 14:46:04.894474 6691 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:04.894529 6691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:04.894694 6691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:04.896693 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "bb83c10bebeb46f68918999b47f12f04"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "238cc74d69053922f63c22030c0c259d"
server_key_iv: "5e77aad0329133c3b0a0a11a000353d5"
server_key_version: "encryptionkey@0"
I20260923 14:46:04.897177 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "bb83c10bebeb46f68918999b47f12f04"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "238cc74d69053922f63c22030c0c259d"
server_key_iv: "5e77aad0329133c3b0a0a11a000353d5"
server_key_version: "encryptionkey@0"
I20260923 14:46:04.900687 6691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:46:04.902895 6706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:04.904090 6691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:46:04.904209 6691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "bb83c10bebeb46f68918999b47f12f04"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "238cc74d69053922f63c22030c0c259d"
server_key_iv: "5e77aad0329133c3b0a0a11a000353d5"
server_key_version: "encryptionkey@0"
I20260923 14:46:04.904335 6691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:04.922921 6691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:04.926088 6691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:04.926277 6691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:04.926851 6691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:04.927637 6691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:04.927712 6691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:04.927819 6691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:04.927872 6691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:04.936476 6691 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:35265
I20260923 14:46:04.936502 6819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:35265 every 8 connection(s)
I20260923 14:46:04.937301 6691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:46:04.941845 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 6691
I20260923 14:46:04.941985 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:46:04.942322 30731 external_mini_cluster.cc:1428] Setting key 09a6ed67432f1308dc16082926260fb7
Sep 23 14:46:04 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174764, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:46:04.951135 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:04.939050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:46205 (local address 127.30.2.254:41823)
0923 14:46:04.939228 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:04.939233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:04.940111 (+ 878us) server_negotiation.cc:407] Connection header received
0923 14:46:04.941415 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:04.941417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:04.941464 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:04.941534 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:04.943109 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:04.943802 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:04.944996 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:04.945231 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:04.945340 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:04.947661 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:04.947664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:04.947666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:04.947688 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:04.949159 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:04.949671 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:04.949676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:04.949678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:04.949761 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:04.950090 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:04.950095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:04.950097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:04.950274 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:04.950380 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:04.950852 (+ 472us) server_negotiation.cc:299] Negotiation successful
0923 14:46:04.950965 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:46:04.951980 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:04.939470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:46205)
0923 14:46:04.939963 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:04.940004 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:04.941214 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:04.941783 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:04.941793 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:04.942326 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:04.942949 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:04.942968 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:04.943952 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:04.943955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:04.944871 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:04.944881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:04.945379 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:04.945383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:04.945543 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:04.946115 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:04.946130 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:04.947496 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:04.949300 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:04.949306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:04.949320 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:04.949569 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:04.949857 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:04.949861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:04.949863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:04.949995 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:04.950379 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:04.950385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:04.950613 (+ 228us) client_negotiation.cc:769] Sending connection context
0923 14:46:04.950796 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 14:46:04.951093 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260923 14:46:04.953101 6820 heartbeater.cc:347] Connected to a master server at 127.30.2.254:41823
I20260923 14:46:04.953346 6820 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:04.953800 6820 heartbeater.cc:511] Master 127.30.2.254:41823 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
I20260923 14:46:04.954849 6495 ts_manager.cc:194] Registered new tserver with Master: bb83c10bebeb46f68918999b47f12f04 (127.30.2.194:35265)
I20260923 14:46:04.955418 6495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:46205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:46:04.971266 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:41823
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:46:05.083211 6825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:05.083520 6825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:05.083595 6825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:05.088034 6825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:05.088121 6825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:05.088213 6825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:46:05.093570 6825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:41823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:05.094671 6825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:05.095759 6825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:05.105788 6831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:05.105782 6830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:05.105782 6833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:05.106156 6825 server_base.cc:1034] running on GCE node
I20260923 14:46:05.106611 6825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:05.107394 6825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:05.108593 6825 hybrid_clock.cc:648] HybridClock initialized: now 1790174765108570 us; error 35 us; skew 500 ppm
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174765, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:05.111536 6825 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:46:05.112462 6825 webserver.cc:460] Webserver started at http://127.30.2.195:42707/ using document root <none> and password file <none>
I20260923 14:46:05.113031 6825 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:05.113091 6825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:05.113329 6825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:05.115347 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "448563a2982b48799c9cce0600f18ff4"
format_stamp: "Formatted at 2026-09-23 14:46:05 on dist-test-slave-wjf2"
server_key: "475a6e693d875976eb47cc3add32f184"
server_key_iv: "c4c3dabf51726c318f07eac6fdf0ca3c"
server_key_version: "encryptionkey@0"
I20260923 14:46:05.115959 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "448563a2982b48799c9cce0600f18ff4"
format_stamp: "Formatted at 2026-09-23 14:46:05 on dist-test-slave-wjf2"
server_key: "475a6e693d875976eb47cc3add32f184"
server_key_iv: "c4c3dabf51726c318f07eac6fdf0ca3c"
server_key_version: "encryptionkey@0"
I20260923 14:46:05.119613 6825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:46:05.122031 6840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:05.123288 6825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:46:05.123435 6825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "448563a2982b48799c9cce0600f18ff4"
format_stamp: "Formatted at 2026-09-23 14:46:05 on dist-test-slave-wjf2"
server_key: "475a6e693d875976eb47cc3add32f184"
server_key_iv: "c4c3dabf51726c318f07eac6fdf0ca3c"
server_key_version: "encryptionkey@0"
I20260923 14:46:05.123545 6825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:05.149487 6825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:05.152866 6825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:05.153119 6825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:05.153746 6825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:05.154596 6825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:05.154644 6825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:05.154724 6825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:05.154775 6825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:05.164142 6825 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:42473
I20260923 14:46:05.164173 6953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:42473 every 8 connection(s)
I20260923 14:46:05.165110 6825 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:46:05.167313 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 6825
I20260923 14:46:05.167452 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:46:05.167747 30731 external_mini_cluster.cc:1428] Setting key 6d70444317ad735cc16de610f718dbae
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174765, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:46:05.178335 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:05.166965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:40115 (local address 127.30.2.254:41823)
0923 14:46:05.167117 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:05.167123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:05.168124 (+ 1001us) server_negotiation.cc:407] Connection header received
0923 14:46:05.169495 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:05.169499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:05.169552 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:05.169630 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:05.170787 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.171322 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.172523 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.172742 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.172850 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.175123 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:05.175126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:05.175128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:05.175150 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:05.176581 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:05.177033 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:05.177036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:05.177037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:05.177107 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:05.177396 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:05.177398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:05.177400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:05.177568 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:05.177640 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:05.178076 (+ 436us) server_negotiation.cc:299] Negotiation successful
0923 14:46:05.178163 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:46:05.179140 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:05.167442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:40115)
0923 14:46:05.167978 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:05.168029 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:05.169289 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:05.169832 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:05.169839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:05.170175 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:05.170629 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.170643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.171455 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.171461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.172396 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.172405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.172886 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.172890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.173053 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.173572 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:05.173587 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:05.174975 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:05.176688 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:05.176693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:05.176704 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:05.176947 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:05.177188 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:05.177190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:05.177192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:05.177316 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:05.177653 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:05.177657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:05.177882 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 14:46:05.178031 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 14:46:05.178280 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":122,"threads_started":1}
I20260923 14:46:05.180526 6954 heartbeater.cc:347] Connected to a master server at 127.30.2.254:41823
I20260923 14:46:05.180804 6954 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:05.181448 6954 heartbeater.cc:511] Master 127.30.2.254:41823 requested a full tablet report, sending...
I20260923 14:46:05.182544 6495 ts_manager.cc:194] Registered new tserver with Master: 448563a2982b48799c9cce0600f18ff4 (127.30.2.195:42473)
I20260923 14:46:05.183183 6495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:40115
I20260923 14:46:05.193692 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab.
Entry for principal oryx/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab.
WARNING: no policy specified for oryx/127.30.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab.
Entry for principal oryx/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab.
WARNING: no policy specified for oryx/127.30.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab.
Entry for principal oryx/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab.
I20260923 14:46:05.271021 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 6557
I20260923 14:46:05.277412 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 6691
I20260923 14:46:05.283340 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 6825
I20260923 14:46:05.289182 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 6463
I20260923 14:46:05.295178 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:41823
--webserver_interface=127.30.2.254
--webserver_port=35057
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:41823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:46:05.404239 6963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:05.404490 6963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:05.404531 6963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:05.408429 6963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:46:05.408479 6963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:05.408499 6963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:05.408515 6963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:46:05.408530 6963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:46:05.413753 6963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:41823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:05.414726 6963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:05.415696 6963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:05.423966 6969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:05.424024 6963 server_base.cc:1034] running on GCE node
W20260923 14:46:05.423991 6968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:05.424155 6971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:05.424667 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:05.425787 6963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:05.426970 6963 hybrid_clock.cc:648] HybridClock initialized: now 1790174765426941 us; error 48 us; skew 500 ppm
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174765, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:05.429675 6963 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:46:05.430577 6963 webserver.cc:460] Webserver started at http://127.30.2.254:35057/ using document root <none> and password file <none>
I20260923 14:46:05.431133 6963 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:05.431254 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:05.435058 6963 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:46:05.437156 6978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:05.438323 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:46:05.438457 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "04c4dcb835f64db1b37da0d4fc84ffcd"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "9b2f9bb2c266d2634ff29757e597e261"
server_key_iv: "d42131a2404c5474299e30430c2358db"
server_key_version: "encryptionkey@0"
I20260923 14:46:05.438978 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:05.465342 6963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:05.468215 6963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:05.468403 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:05.475307 6963 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:41823
I20260923 14:46:05.475335 7030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:41823 every 8 connection(s)
I20260923 14:46:05.476316 6963 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:46:05.480602 7031 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:46:05.481211 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 6963
I20260923 14:46:05.481724 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:33809
--local_ip_for_outbound_sockets=127.30.2.193
--tserver_master_addrs=127.30.2.254:41823
--webserver_port=39153
--webserver_interface=127.30.2.193
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.2.193
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:46:05.482815 7031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd: Bootstrap starting.
I20260923 14:46:05.499464 7031 log.cc:824] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd: Log is configured to *not* fsync() on all Append() calls
I20260923 14:46:05.506346 7031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:46:05.506721 7031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd: Bootstrap complete.
I20260923 14:46:05.510978 7031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } }
I20260923 14:46:05.511433 7031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c4dcb835f64db1b37da0d4fc84ffcd, State: Initialized, Role: FOLLOWER
I20260923 14:46:05.511986 7031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } }
I20260923 14:46:05.512130 7031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:46:05.512221 7031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:46:05.512342 7031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:46:05.514587 7031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } }
I20260923 14:46:05.514950 7031 leader_election.cc:302] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04c4dcb835f64db1b37da0d4fc84ffcd; no voters:
I20260923 14:46:05.515319 7031 leader_election.cc:288] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 14:46:05.515415 7035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 2 FOLLOWER]: Leader election won for term 2
I20260923 14:46:05.515614 7035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [term 2 LEADER]: Becoming Leader. State: Replica: 04c4dcb835f64db1b37da0d4fc84ffcd, State: Running, Role: LEADER
I20260923 14:46:05.516039 7035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } }
I20260923 14:46:05.516662 7031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:46:05.517426 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } } }
I20260923 14:46:05.517549 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [sys.catalog]: This master's current role is: LEADER
I20260923 14:46:05.518447 7043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:46:05.518587 7037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c4dcb835f64db1b37da0d4fc84ffcd. Latest consensus state: current_term: 2 leader_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4dcb835f64db1b37da0d4fc84ffcd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 41823 } } }
I20260923 14:46:05.518762 7037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd [sys.catalog]: This master's current role is: LEADER
I20260923 14:46:05.521698 7043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:46:05.522475 7043 catalog_manager.cc:1259] Loaded cluster ID: 3c7a255c66a54c3590692e9e4c13b14e
I20260923 14:46:05.522535 7043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:46:05.524930 7043 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:46:05.525606 7043 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 04c4dcb835f64db1b37da0d4fc84ffcd: Loaded TSK: 0
I20260923 14:46:05.526856 7043 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:46:05.595506 7033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:05.595793 7033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:05.595897 7033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:05.600024 7033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:05.600100 7033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:05.600191 7033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:46:05.605405 7033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.193.keytab
--principal=oryx/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:33809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=39153
--tserver_master_addrs=127.30.2.254:41823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:05.606421 7033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:05.607532 7033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:05.617519 7058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:05.617524 7057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:05.617748 7060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:05.618037 7033 server_base.cc:1034] running on GCE node
I20260923 14:46:05.618458 7033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:05.619144 7033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:05.620337 7033 hybrid_clock.cc:648] HybridClock initialized: now 1790174765620306 us; error 46 us; skew 500 ppm
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174765, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:05.623147 7033 init.cc:377] Logged in from keytab as oryx/127.30.2.193@KRBTEST.COM (short username oryx)
I20260923 14:46:05.624135 7033 webserver.cc:460] Webserver started at http://127.30.2.193:39153/ using document root <none> and password file <none>
I20260923 14:46:05.624686 7033 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:05.624800 7033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:05.628638 7033 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:46:05.630923 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:05.632295 7033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:46:05.632414 7033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "9d1ef168d26a46f4bb806c14862dc80b"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "7e1ebb78998bdc92d6a952af87d8db06"
server_key_iv: "87b2a2a803ba94c97f1d6c57e9438991"
server_key_version: "encryptionkey@0"
I20260923 14:46:05.632956 7033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:05.654649 7033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:05.658021 7033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:05.658236 7033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:05.658843 7033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:05.659765 7033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:05.659854 7033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:05.659929 7033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:05.659971 7033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:05.668670 7033 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:33809
I20260923 14:46:05.668704 7180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:33809 every 8 connection(s)
I20260923 14:46:05.669710 7033 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:46:05.678493 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 7033
I20260923 14:46:05.678990 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:35265
--local_ip_for_outbound_sockets=127.30.2.194
--tserver_master_addrs=127.30.2.254:41823
--webserver_port=41415
--webserver_interface=127.30.2.194
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.2.194
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 14:46:05.682652 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:05.671594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:37775 (local address 127.30.2.254:41823)
0923 14:46:05.672057 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:05.672076 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:05.672376 (+ 300us) server_negotiation.cc:407] Connection header received
0923 14:46:05.673934 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:05.673944 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:05.674233 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:05.674551 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:05.675719 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.676725 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.677934 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.678410 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.678595 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.681960 (+ 3365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:37775: BlockingRecv error: recv got EOF from 127.30.2.193:37775 (error 108)
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":146,"threads_started":1}
I20260923 14:46:05.682662 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:05.671753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:37775)
0923 14:46:05.672224 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:05.672269 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:05.673510 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:05.674627 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:05.674635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:05.675044 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:05.675539 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.675554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.676896 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.676899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.677807 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.677817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.678575 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.678579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.678765 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.679347 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:05.679364 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:05.681728 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":103,"threads_started":1}
W20260923 14:46:05.683089 7181 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:41823 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.2.254:41823: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:05.688719 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:05.683559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:58467)
0923 14:46:05.683665 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:05.683682 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:05.683758 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:05.684027 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:05.684030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:05.684178 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:05.684258 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.684269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.684957 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.684962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.685624 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.685635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.686105 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.686109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.686207 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.686695 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:05.686706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:05.688551 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 14:46:05.688759 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:05.683631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:58467 (local address 127.30.2.254:41823)
0923 14:46:05.683736 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:05.683740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:05.683751 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:46:05.683874 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:05.683876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:05.683925 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:05.684015 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:05.684373 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.684852 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.685774 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.685993 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.686102 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.688677 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:58467: BlockingRecv error: recv got EOF from 127.30.2.193:58467 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:05.694492 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:05.689249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:34223)
0923 14:46:05.689354 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:05.689368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:05.689444 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:05.689736 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:05.689739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:05.689882 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:05.689947 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.689952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.690641 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.690646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.691150 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.691157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.691623 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.691626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.691757 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.692355 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:05.692369 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:05.694332 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 14:46:05.694603 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:05.689302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:34223 (local address 127.30.2.254:41823)
0923 14:46:05.689406 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:05.689410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:05.689423 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:46:05.689594 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:05.689597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:05.689641 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:05.689722 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:05.690053 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.690529 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.691267 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.691481 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.691620 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.694462 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:34223: BlockingRecv error: recv got EOF from 127.30.2.193:34223 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260923 14:46:05.694922 7181 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:46:05.790514 7185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:05.790802 7185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:05.790877 7185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:05.794883 7185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:05.794963 7185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:05.795051 7185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:46:05.800168 7185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.194.keytab
--principal=oryx/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:35265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=41415
--tserver_master_addrs=127.30.2.254:41823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:05.801198 7185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:05.802223 7185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:05.811934 7193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:05.811940 7191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:05.811940 7190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:05.812005 7185 server_base.cc:1034] running on GCE node
I20260923 14:46:05.812580 7185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:05.813308 7185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:05.814504 7185 hybrid_clock.cc:648] HybridClock initialized: now 1790174765814490 us; error 25 us; skew 500 ppm
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174765, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:05.817514 7185 init.cc:377] Logged in from keytab as oryx/127.30.2.194@KRBTEST.COM (short username oryx)
I20260923 14:46:05.818446 7185 webserver.cc:460] Webserver started at http://127.30.2.194:41415/ using document root <none> and password file <none>
I20260923 14:46:05.819039 7185 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:05.819159 7185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:05.823298 7185 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:46:05.825798 7200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:05.827064 7185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:46:05.827224 7185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "bb83c10bebeb46f68918999b47f12f04"
format_stamp: "Formatted at 2026-09-23 14:46:04 on dist-test-slave-wjf2"
server_key: "238cc74d69053922f63c22030c0c259d"
server_key_iv: "5e77aad0329133c3b0a0a11a000353d5"
server_key_version: "encryptionkey@0"
I20260923 14:46:05.827766 7185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:05.847509 7185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:05.850745 7185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:05.850956 7185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:05.851555 7185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:05.852392 7185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:05.852465 7185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:05.852538 7185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:05.852582 7185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:05.861549 7185 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:35265
I20260923 14:46:05.861575 7313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:35265 every 8 connection(s)
I20260923 14:46:05.862496 7185 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:46:05.865407 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 7185
I20260923 14:46:05.865994 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:42473
--local_ip_for_outbound_sockets=127.30.2.195
--tserver_master_addrs=127.30.2.254:41823
--webserver_port=42707
--webserver_interface=127.30.2.195
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.2.195
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 14:46:05.874469 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:05.864305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:47765 (local address 127.30.2.254:41823)
0923 14:46:05.864477 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:05.864481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:05.865394 (+ 913us) server_negotiation.cc:407] Connection header received
0923 14:46:05.867466 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:05.867470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:05.867543 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:05.867644 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:05.868862 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.869420 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.870545 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.870773 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.870884 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.874348 (+ 3464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:47765: BlockingRecv error: recv got EOF from 127.30.2.194:47765 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 14:46:05.875047 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:05.864681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:47765)
0923 14:46:05.865191 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:05.865256 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:05.867182 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:05.867875 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:05.867881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:05.868229 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:05.868709 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.868724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.869548 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.869552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.870416 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.870426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.870921 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.870926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.871085 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.871658 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:05.871674 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:05.874225 (+ 2551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":98,"threads_started":1}
W20260923 14:46:05.875619 7314 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:41823 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.2.254:41823: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:05.881102 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:05.876090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:54887)
0923 14:46:05.876222 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:05.876237 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:05.876311 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:05.876558 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:05.876561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:05.876722 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:05.876807 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.876812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.877512 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.877517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.878067 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.878074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.878513 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.878517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.878605 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.879138 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:05.879149 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:05.880960 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260923 14:46:05.881457 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:05.876236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:54887 (local address 127.30.2.254:41823)
0923 14:46:05.876367 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:05.876374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:05.876388 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:46:05.876424 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:05.876426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:05.876460 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:05.876544 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:05.876919 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.877404 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.878193 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.878404 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.878510 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.881072 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:54887: BlockingRecv error: recv got EOF from 127.30.2.194:54887 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:05 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:05.887495 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:05.881660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:52415)
0923 14:46:05.881791 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:05.881803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:05.881892 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:05.882322 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:05.882325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:05.882440 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:05.882510 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.882514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.883554 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.883557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.884138 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:05.884147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.884769 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.884772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:05.884835 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.885343 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:05.885355 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:05.887394 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 14:46:05.887770 7314 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:46:05.887876 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:05.881938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:52415 (local address 127.30.2.254:41823)
0923 14:46:05.882087 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:05.882092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:05.882107 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:46:05.882152 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:05.882155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:05.882201 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:05.882285 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:05.882610 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.883427 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.884261 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:05.884626 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:05.884746 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:05.887581 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:52415: BlockingRecv error: recv got EOF from 127.30.2.194:52415 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260923 14:46:05.982959 7317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:05.983213 7317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:05.983253 7317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:05.987145 7317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:05.987195 7317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:05.987249 7317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:46:05.992326 7317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/krb5kdc/oryx_127.30.2.195.keytab
--principal=oryx/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:42473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=42707
--tserver_master_addrs=127.30.2.254:41823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:05.993299 7317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:05.994287 7317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:06.003602 7325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:06.003589 7322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:06.003610 7323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:06.003825 7317 server_base.cc:1034] running on GCE node
I20260923 14:46:06.004454 7317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:06.005064 7317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:06.006273 7317 hybrid_clock.cc:648] HybridClock initialized: now 1790174766006250 us; error 30 us; skew 500 ppm
Sep 23 14:46:06 dist-test-slave-wjf2 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174766, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:06.009249 7317 init.cc:377] Logged in from keytab as oryx/127.30.2.195@KRBTEST.COM (short username oryx)
I20260923 14:46:06.009977 7317 webserver.cc:460] Webserver started at http://127.30.2.195:42707/ using document root <none> and password file <none>
I20260923 14:46:06.010476 7317 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:06.010542 7317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:06.014338 7317 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:46:06.016402 7332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:06.017901 7317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:46:06.018015 7317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "448563a2982b48799c9cce0600f18ff4"
format_stamp: "Formatted at 2026-09-23 14:46:05 on dist-test-slave-wjf2"
server_key: "475a6e693d875976eb47cc3add32f184"
server_key_iv: "c4c3dabf51726c318f07eac6fdf0ca3c"
server_key_version: "encryptionkey@0"
I20260923 14:46:06.018565 7317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:06.034219 7317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:06.037367 7317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:06.037563 7317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:06.038157 7317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:06.038955 7317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:06.039000 7317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:06.039067 7317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:06.039106 7317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:06.048238 7317 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:42473
I20260923 14:46:06.048345 7445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:42473 every 8 connection(s)
I20260923 14:46:06.049229 7317 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:46:06.053166 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 7317
Sep 23 14:46:06 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:06 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 14:46:06.061954 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:06.051113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:45567 (local address 127.30.2.254:41823)
0923 14:46:06.051257 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:06.051261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:06.052208 (+ 947us) server_negotiation.cc:407] Connection header received
0923 14:46:06.053578 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:06.053582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:06.053651 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:06.053735 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:06.055414 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.056319 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.057857 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.058109 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.058187 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:06.061826 (+ 3639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:45567: BlockingRecv error: recv got EOF from 127.30.2.195:45567 (error 108)
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":60}
I20260923 14:46:06.062553 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:06.051479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:45567)
0923 14:46:06.052055 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:06.052095 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:06.053329 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:06.053988 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:06.053999 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:06.054771 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:06.055272 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:06.055285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.056467 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.056471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:06.057746 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:06.057756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.058278 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.058283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:06.058444 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:06.058956 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:06.058980 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:06.061732 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":110,"threads_started":1}
W20260923 14:46:06.063073 7446 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:41823 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.2.254:41823: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
I20260923 14:46:06.064447 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:06.054272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.30.2.254:41823)
0923 14:46:06.054485 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:06.054489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:06.054506 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:46:06.054545 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:06.054548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:06.054591 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:06.054690 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:06.055527 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.056319 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.057560 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.057794 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.057917 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:06.059649 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:06.059654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:06.059668 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:06.059695 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:06.061849 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:06.062331 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:06.062339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:06.062350 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:06.062462 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:06.063139 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:06.063145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:06.063147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:06.063594 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:06.063807 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:06.064109 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 14:46:06.064335 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
Sep 23 14:46:06 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:06 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:06.069054 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:06.063517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:42611)
0923 14:46:06.063620 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:06.063643 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:06.063719 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:06.064000 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:06.064003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:06.064150 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:06.064286 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:06.064294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.065188 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.065191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:06.065902 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:06.065912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.066392 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.066396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:06.066724 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:06.067164 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:06.067179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:06.068969 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 14:46:06.069423 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:06.063593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:42611 (local address 127.30.2.254:41823)
0923 14:46:06.063699 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:06.063703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:06.063714 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:46:06.063868 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:06.063870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:06.063904 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:06.063984 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:06.064397 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.065077 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.066015 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.066255 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.066358 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:06.069108 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:42611: BlockingRecv error: recv got EOF from 127.30.2.195:42611 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 23 14:46:06 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:06 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:06.073925 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:06.069596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:41461)
0923 14:46:06.069688 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:06.069702 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:06.069776 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:06.069959 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:06.069961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:06.070078 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:06.070146 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:06.070151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.070871 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.070873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:06.071494 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:06.071502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.071948 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.071950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:06.072006 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:06.072362 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:06.072373 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:06.073847 (+ 1474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 14:46:06.074045 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:06.069651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:41461 (local address 127.30.2.254:41823)
0923 14:46:06.069776 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:06.069780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:06.069791 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:46:06.069837 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:06.069839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:06.069869 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:06.069929 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:06.070312 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.070778 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.071595 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.071855 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.071933 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:06.073950 (+ 2017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:41461: BlockingRecv error: recv got EOF from 127.30.2.195:41461 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 14:46:06.074154 7446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 14:46:06 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:06 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:06.701746 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:06.695547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:34941)
0923 14:46:06.695940 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:06.695955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:06.696053 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:06.696309 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:06.696311 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:06.696498 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:06.696625 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:06.696630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.697434 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.697437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:06.698087 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:06.698096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.698552 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.698570 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:06.698644 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:06.699320 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:06.699333 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:06.701540 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":89,"threads_started":1}
W20260923 14:46:06.701807 7452 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:06.695707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:34941 (local address 127.30.2.254:41823)
0923 14:46:06.695991 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:06.695995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:06.696009 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:46:06.696139 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:06.696142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:06.696187 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:06.696268 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:06.696738 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.697319 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.698205 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.698449 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.698534 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:06.701678 (+ 3144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:34941: BlockingRecv error: recv got EOF from 127.30.2.193:34941 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":114,"threads_started":1}
Sep 23 14:46:06 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:06 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:06.893997 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:06.888459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:59479)
0923 14:46:06.888780 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:06.888791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:06.888919 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:06.889195 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:06.889198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:06.889402 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:06.889506 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:06.889512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.890282 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.890288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:06.890872 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:06.890881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.891356 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.891360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:06.891460 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:06.892020 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:06.892031 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:06.893813 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":123,"threads_started":1}
W20260923 14:46:06.894055 7452 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:06.888513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:59479 (local address 127.30.2.254:41823)
0923 14:46:06.888691 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:06.888696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:06.888863 (+ 167us) server_negotiation.cc:407] Connection header received
0923 14:46:06.889025 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:06.889028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:06.889083 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:06.889175 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:06.889624 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.890170 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.890995 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:06.891241 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:06.891352 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:06.893947 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:59479: BlockingRecv error: recv got EOF from 127.30.2.194:59479 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 14:46:07 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:07 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:07.080642 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:07.074911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:46717)
0923 14:46:07.075261 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:07.075277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:07.075380 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:07.075621 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:07.075623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:07.075867 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:07.075980 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:07.075987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.076703 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.076706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:07.077365 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:07.077374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.077799 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.077802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:07.077874 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:07.078519 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:07.078533 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:07.080512 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":107,"threads_started":1}
W20260923 14:46:07.080912 7452 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:07.074969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:46717 (local address 127.30.2.254:41823)
0923 14:46:07.075119 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:07.075125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:07.075374 (+ 249us) server_negotiation.cc:407] Connection header received
0923 14:46:07.075451 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:07.075453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:07.075503 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:07.075577 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:07.076081 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.076600 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.077475 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.077690 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.077770 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:07.080656 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:46717: BlockingRecv error: recv got EOF from 127.30.2.195:46717 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 14:46:07 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:07 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:07.708118 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:07.702593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:41301)
0923 14:46:07.702897 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:07.702912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:07.703006 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:07.703257 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:07.703259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:07.703436 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:07.703530 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:07.703535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.704373 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.704376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:07.704957 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:07.704966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.705429 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.705432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:07.705499 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:07.706018 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:07.706030 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:07.707935 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":68,"threads_started":1}
W20260923 14:46:07.708185 7456 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:07.702732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:41301 (local address 127.30.2.254:41823)
0923 14:46:07.702973 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:07.702976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:07.702989 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:46:07.703093 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:07.703095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:07.703145 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:07.703219 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:07.703628 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.704244 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.705078 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.705320 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.705399 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:07.708056 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:41301: BlockingRecv error: recv got EOF from 127.30.2.193:41301 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
Sep 23 14:46:07 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:07 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:07.900563 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:07.894778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:58797)
0923 14:46:07.895059 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:07.895073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:07.895207 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:07.895457 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:07.895459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:07.895638 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:07.895735 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:07.895740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.896798 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.896805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:07.897431 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:07.897442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.897896 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.897900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:07.897974 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:07.898528 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:07.898544 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:07.900373 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
W20260923 14:46:07.900592 7456 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:07.894844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:58797 (local address 127.30.2.254:41823)
0923 14:46:07.894972 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:07.894976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:07.895173 (+ 197us) server_negotiation.cc:407] Connection header received
0923 14:46:07.895300 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:07.895302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:07.895350 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:07.895411 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:07.895886 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.896666 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.897544 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:07.897763 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:07.897843 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:07.900494 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:58797: BlockingRecv error: recv got EOF from 127.30.2.194:58797 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 23 14:46:08 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:08 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:08.092017 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:08.081654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:53215)
0923 14:46:08.081972 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:08.081986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:08.082087 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:08.082352 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:08.082354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:08.082527 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:08.082631 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:08.082636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.083395 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.083398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:08.083987 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:08.083995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.084427 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.084430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:08.084496 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:08.089618 (+ 5122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:08.089630 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:08.091820 (+ 2190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
W20260923 14:46:08.092168 7456 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:08.081776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:53215 (local address 127.30.2.254:41823)
0923 14:46:08.081966 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:08.081969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:08.082037 (+ 68us) server_negotiation.cc:407] Connection header received
0923 14:46:08.082170 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:08.082172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:08.082217 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:08.082281 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:08.082754 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.083273 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.084099 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.084321 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.084403 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:08.091971 (+ 7568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:53215: BlockingRecv error: recv got EOF from 127.30.2.195:53215 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 23 14:46:08 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:08 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:08.715297 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:08.708998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:41641)
0923 14:46:08.709285 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:08.709298 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:08.709389 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:08.709690 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:08.709693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:08.709912 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:08.710013 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:08.710018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.711198 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.711201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:08.711760 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:08.711767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.712357 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.712361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:08.712435 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:08.713189 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:08.713204 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:08.715162 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
W20260923 14:46:08.715422 7460 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:08.709100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:41641 (local address 127.30.2.254:41823)
0923 14:46:08.709400 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:08.709405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:08.709425 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:46:08.709464 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:08.709467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:08.709510 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:08.709633 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:08.710162 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.711094 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.711923 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.712203 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.712287 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:08.715287 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:41641: BlockingRecv error: recv got EOF from 127.30.2.193:41641 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":129,"threads_started":1}
Sep 23 14:46:08 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:08 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:08.907286 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:08.901377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:36869)
0923 14:46:08.901730 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:08.901753 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:08.901893 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:08.902197 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:08.902200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:08.902401 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:08.902497 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:08.902503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.903279 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.903284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:08.903899 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:08.903908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.904373 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.904377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:08.904462 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:08.905039 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:08.905052 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:08.907125 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":109,"threads_started":1}
W20260923 14:46:08.907550 7460 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:08.901519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:36869 (local address 127.30.2.254:41823)
0923 14:46:08.901697 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:08.901702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:08.901829 (+ 127us) server_negotiation.cc:407] Connection header received
0923 14:46:08.901993 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:08.901997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:08.902051 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:08.902140 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:08.902618 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.903165 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.904022 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:08.904257 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:08.904370 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:08.907319 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:36869: BlockingRecv error: recv got EOF from 127.30.2.194:36869 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 23 14:46:09 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:09 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:09.098465 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:09.092810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:42119)
0923 14:46:09.093082 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:09.093097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:09.093235 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:09.093516 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:09.093519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:09.093738 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:09.093837 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:09.093843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.094611 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.094615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:09.095206 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:09.095215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.095693 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.095696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:09.095818 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:09.096391 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:09.096406 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:09.098239 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
W20260923 14:46:09.098500 7460 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:09.092872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:42119 (local address 127.30.2.254:41823)
0923 14:46:09.093027 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:09.093033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:09.093182 (+ 149us) server_negotiation.cc:407] Connection header received
0923 14:46:09.093341 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:09.093345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:09.093397 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:09.093490 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:09.093955 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.094499 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.095328 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.095572 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.095686 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:09.098387 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:42119: BlockingRecv error: recv got EOF from 127.30.2.195:42119 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 14:46:09 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:09 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:09.722353 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:09.716184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:50193)
0923 14:46:09.716463 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:09.716476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:09.716609 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:09.716924 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:09.716927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:09.717103 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:09.717253 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:09.717261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.718333 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.718336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:09.718922 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:09.718931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.719436 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.719439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:09.719651 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:09.720327 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:09.720339 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:09.722182 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
W20260923 14:46:09.722594 7464 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:09.716262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:50193 (local address 127.30.2.254:41823)
0923 14:46:09.716542 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:09.716570 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:09.716610 (+ 40us) server_negotiation.cc:407] Connection header received
0923 14:46:09.716744 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:09.716748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:09.716807 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:09.716894 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:09.717444 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.718176 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.719029 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.719295 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.719386 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:09.722340 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:50193: BlockingRecv error: recv got EOF from 127.30.2.193:50193 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
Sep 23 14:46:09 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:09 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:09.913411 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:09.908144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:49435)
0923 14:46:09.908428 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:09.908441 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:09.908538 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:09.908783 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:09.908785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:09.908959 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:09.909056 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:09.909062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.909812 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.909815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:09.910379 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:09.910389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.910803 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.910806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:09.910870 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:09.911357 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:09.911369 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:09.913297 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":102,"threads_started":1}
W20260923 14:46:09.913560 7464 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:09.908156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:49435 (local address 127.30.2.254:41823)
0923 14:46:09.908309 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:09.908312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:09.908509 (+ 197us) server_negotiation.cc:407] Connection header received
0923 14:46:09.908619 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:09.908621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:09.908678 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:09.908748 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:09.909158 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.909685 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.910487 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:09.910703 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:09.910777 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:09.913440 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:49435: BlockingRecv error: recv got EOF from 127.30.2.194:49435 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 14:46:10 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:10 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:10.105124 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:10.099275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:50681)
0923 14:46:10.099641 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:10.099659 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:10.099841 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:10.100158 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:10.100161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:10.100373 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:10.100470 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:10.100475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.101253 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.101258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:10.101828 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:10.101836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.102314 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.102318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:10.102409 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:10.102986 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:10.102999 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:10.104941 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":110,"threads_started":1}
W20260923 14:46:10.105172 7464 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:10.099433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:50681 (local address 127.30.2.254:41823)
0923 14:46:10.099643 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:10.099649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:10.099749 (+ 100us) server_negotiation.cc:407] Connection header received
0923 14:46:10.099947 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:10.099950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:10.100008 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:10.100095 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:10.100601 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.101138 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.101955 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.102196 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.102310 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:10.105066 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:50681: BlockingRecv error: recv got EOF from 127.30.2.195:50681 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 23 14:46:10 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:10 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:10.729219 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:10.723203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:53843)
0923 14:46:10.723535 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:10.723548 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:10.723642 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:10.723933 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:10.723935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:10.724114 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:10.724271 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:10.724282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.725426 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.725431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:10.726034 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:10.726044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.726511 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.726520 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:10.726584 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:10.727245 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:10.727258 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:10.729084 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
W20260923 14:46:10.729431 7467 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:10.723269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:53843 (local address 127.30.2.254:41823)
0923 14:46:10.723562 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:10.723565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:10.723600 (+ 35us) server_negotiation.cc:407] Connection header received
0923 14:46:10.723739 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:10.723742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:10.723812 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:10.723904 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:10.724411 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.725308 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.726152 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.726385 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.726477 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:10.729231 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:53843: BlockingRecv error: recv got EOF from 127.30.2.193:53843 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
Sep 23 14:46:10 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:10 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:10.920163 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:10.914314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:39659)
0923 14:46:10.914613 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:10.914626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:10.914712 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:10.915010 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:10.915012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:10.915201 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:10.915295 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:10.915300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.916169 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.916173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:10.916857 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:10.916866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.917303 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.917306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:10.917386 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:10.917897 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:10.917908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:10.920013 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
W20260923 14:46:10.920310 7467 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:10.914431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:39659 (local address 127.30.2.254:41823)
0923 14:46:10.914719 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:10.914723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:10.914769 (+ 46us) server_negotiation.cc:407] Connection header received
0923 14:46:10.914809 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:10.914811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:10.914870 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:10.914939 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:10.915418 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.916019 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.916960 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:10.917195 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:10.917275 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:10.920162 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:39659: BlockingRecv error: recv got EOF from 127.30.2.194:39659 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 14:46:11 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:11 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:11.111908 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:11.106075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:48501)
0923 14:46:11.106434 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:11.106453 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:11.106585 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:11.106884 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:11.106887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:11.107108 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:11.107209 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:11.107215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.108072 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.108077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:11.108685 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:11.108693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.109202 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.109206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:11.109302 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:11.109787 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:11.109799 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:11.111714 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":96,"threads_started":1}
W20260923 14:46:11.112044 7467 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:11.106217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:48501 (local address 127.30.2.254:41823)
0923 14:46:11.106392 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:11.106397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:11.106530 (+ 133us) server_negotiation.cc:407] Connection header received
0923 14:46:11.106699 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:11.106702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:11.106752 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:11.106860 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:11.107330 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.107964 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.108809 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.109059 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.109189 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:11.111904 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:48501: BlockingRecv error: recv got EOF from 127.30.2.195:48501 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 23 14:46:11 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:11 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:11.736089 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:11.729978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:52461)
0923 14:46:11.730271 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:11.730306 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:11.730423 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:11.730794 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:11.730797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:11.731062 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:11.731204 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:11.731211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.732212 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.732216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:11.732801 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:11.732809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.733290 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.733294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:11.733378 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:11.733915 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:11.733927 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:11.735956 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
W20260923 14:46:11.736243 7472 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:11.730057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:52461 (local address 127.30.2.254:41823)
0923 14:46:11.730295 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:11.730299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:11.730382 (+ 83us) server_negotiation.cc:407] Connection header received
0923 14:46:11.730538 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:11.730544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:11.730629 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:11.730725 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:11.731330 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.732099 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.732920 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.733166 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.733275 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:11.736085 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:52461: BlockingRecv error: recv got EOF from 127.30.2.193:52461 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
Sep 23 14:46:11 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:11 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:11.926544 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:11.921010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:35043)
0923 14:46:11.921270 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:11.921285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:11.921393 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:11.921703 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:11.921706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:11.921973 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:11.922078 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:11.922085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.922958 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.922962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:11.923553 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:11.923563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.924058 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.924061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:11.924124 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:11.924602 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:11.924614 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:11.926431 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
W20260923 14:46:11.926695 7472 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:11.921094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:35043 (local address 127.30.2.254:41823)
0923 14:46:11.921258 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:11.921263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:11.921363 (+ 100us) server_negotiation.cc:407] Connection header received
0923 14:46:11.921518 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:11.921521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:11.921585 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:11.921660 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:11.922209 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.922817 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.923672 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:11.923961 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:11.924083 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:11.926572 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:35043: BlockingRecv error: recv got EOF from 127.30.2.194:35043 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 14:46:12 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:12 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:12.119261 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:12.112786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:33537)
0923 14:46:12.113093 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:12.113108 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:12.113225 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:12.113478 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:12.113480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:12.113696 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:12.113798 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:12.113803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.114909 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.114913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:12.115860 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:12.115872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.116329 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.116333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:12.116406 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:12.116986 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:12.116999 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:12.119138 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":110,"threads_started":1}
W20260923 14:46:12.119437 7472 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:12.112908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:33537 (local address 127.30.2.254:41823)
0923 14:46:12.113047 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:12.113051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:12.113209 (+ 158us) server_negotiation.cc:407] Connection header received
0923 14:46:12.113305 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:12.113307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:12.113357 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:12.113432 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:12.113952 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.114741 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.115978 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.116199 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.116279 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:12.119282 (+ 3003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:33537: BlockingRecv error: recv got EOF from 127.30.2.195:33537 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 14:46:12 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:12 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:12.743263 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:12.737036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:51853)
0923 14:46:12.737424 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:12.737439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:12.737547 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:12.737826 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:12.737828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:12.738031 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:12.738168 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:12.738174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.738951 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.738954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:12.739653 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:12.739663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.740262 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.740265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:12.740332 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:12.741003 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:12.741016 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:12.743144 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":67,"threads_started":1}
W20260923 14:46:12.743454 7476 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:12.737138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:51853 (local address 127.30.2.254:41823)
0923 14:46:12.737452 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:12.737456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:12.737492 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:46:12.737657 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:12.737659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:12.737705 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:12.737782 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:12.738276 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.738821 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.739796 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.740053 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.740138 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:12.743283 (+ 3145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:51853: BlockingRecv error: recv got EOF from 127.30.2.193:51853 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
Sep 23 14:46:12 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:12 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:12.933027 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:12.927437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:57183)
0923 14:46:12.927747 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:12.927761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:12.927878 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:12.928104 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:12.928106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:12.928306 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:12.928403 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:12.928408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.929388 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.929391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:12.929937 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:12.929945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.930406 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.930408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:12.930467 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:12.930966 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:12.930978 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:12.932893 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
W20260923 14:46:12.933213 7476 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:12.927496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:57183 (local address 127.30.2.254:41823)
0923 14:46:12.927667 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:12.927671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:12.927844 (+ 173us) server_negotiation.cc:407] Connection header received
0923 14:46:12.927955 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:12.927957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:12.928001 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:12.928068 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:12.928501 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.929270 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.930035 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:12.930304 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:12.930384 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:12.933043 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:57183: BlockingRecv error: recv got EOF from 127.30.2.194:57183 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 14:46:13 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:13 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:13.130666 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:13.120034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:38695)
0923 14:46:13.120368 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:13.120385 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:13.120508 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:13.120925 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:13.120929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:13.121209 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:13.121342 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:13.121348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.122166 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.122172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:13.122735 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:13.122744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.123190 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.123193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:13.123290 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:13.128348 (+ 5058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:13.128369 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:13.130530 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":66,"threads_started":1}
W20260923 14:46:13.130739 7476 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:13.120145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:38695 (local address 127.30.2.254:41823)
0923 14:46:13.120349 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:13.120352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:13.120462 (+ 110us) server_negotiation.cc:407] Connection header received
0923 14:46:13.120620 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:13.120625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:13.120686 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:13.120803 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:13.121464 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.122058 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.122858 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.123078 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.123186 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:13.130649 (+ 7463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:38695: BlockingRecv error: recv got EOF from 127.30.2.195:38695 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 14:46:13 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:13 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:13.749380 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:13.744038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:45951)
0923 14:46:13.744327 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:13.744341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:13.744435 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:13.744732 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:13.744735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:13.744913 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:13.745011 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:13.745016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.745780 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.745783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:13.746394 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:13.746402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.746839 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.746842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:13.746907 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:13.747443 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:13.747455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:13.749262 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":111,"threads_started":1}
W20260923 14:46:13.749521 7480 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:13.744183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:45951 (local address 127.30.2.254:41823)
0923 14:46:13.744412 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:13.744415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:13.744428 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:46:13.744522 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:13.744524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:13.744567 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:13.744702 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:13.745119 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.745669 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.746495 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.746727 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.746814 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:13.749388 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:45951: BlockingRecv error: recv got EOF from 127.30.2.193:45951 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
Sep 23 14:46:13 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:13 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:13.939646 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:13.933793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:49111)
0923 14:46:13.934054 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:13.934069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:13.934174 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:13.934406 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:13.934409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:13.934588 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:13.934702 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:13.934707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.935569 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.935572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:13.936241 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:13.936252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.936656 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.936659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:13.936718 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:13.937355 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:13.937366 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:13.939480 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
W20260923 14:46:13.939709 7480 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:13.933836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:49111 (local address 127.30.2.254:41823)
0923 14:46:13.933972 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:13.933976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:13.934135 (+ 159us) server_negotiation.cc:407] Connection header received
0923 14:46:13.934252 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:13.934254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:13.934299 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:13.934370 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:13.934796 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.935472 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.936340 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:13.936551 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:13.936628 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:13.939593 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:49111: BlockingRecv error: recv got EOF from 127.30.2.194:49111 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 14:46:14 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:14 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:14.137657 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:14.131559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:47641)
0923 14:46:14.131985 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:14.132002 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:14.132181 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:14.132516 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:14.132520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:14.132738 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:14.132832 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:14.132837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.133601 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.133606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:14.134214 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:14.134223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.134683 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.134687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:14.134774 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:14.135401 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:14.135413 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:14.137485 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":142,"threads_started":1}
W20260923 14:46:14.137749 7480 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:14.131639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:47641 (local address 127.30.2.254:41823)
0923 14:46:14.131856 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:14.131861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:14.132104 (+ 243us) server_negotiation.cc:407] Connection header received
0923 14:46:14.132308 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:14.132311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:14.132363 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:14.132479 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:14.132958 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.133487 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.134342 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.134565 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.134678 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:14.137607 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:47641: BlockingRecv error: recv got EOF from 127.30.2.195:47641 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 23 14:46:14 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:14 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:14.756078 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:14.750239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:42491)
0923 14:46:14.750536 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:14.750549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:14.750641 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:14.750876 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:14.750878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:14.751049 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:14.751168 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:14.751173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.751959 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.751962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:14.752519 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:14.752527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.752946 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.752948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:14.753007 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:14.753634 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:14.753646 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:14.755909 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
W20260923 14:46:14.756201 7484 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:14.750351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:42491 (local address 127.30.2.254:41823)
0923 14:46:14.750603 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:14.750607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:14.750619 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:46:14.750723 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:14.750725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:14.750768 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:14.750841 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:14.751280 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.751864 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.752632 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.752851 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.752928 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:14.756056 (+ 3128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:42491: BlockingRecv error: recv got EOF from 127.30.2.193:42491 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":55,"threads_started":1}
Sep 23 14:46:14 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:14 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:14.946210 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:14.940380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:52069)
0923 14:46:14.940660 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:14.940674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:14.940768 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:14.941007 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:14.941009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:14.941189 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:14.941286 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:14.941291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.942159 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.942162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:14.942726 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:14.942735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.943327 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.943330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:14.943394 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:14.943985 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:14.943997 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:14.946024 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
W20260923 14:46:14.946388 7484 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:14.940552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:52069 (local address 127.30.2.254:41823)
0923 14:46:14.940700 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:14.940704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:14.940724 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:46:14.940860 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:14.940862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:14.940908 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:14.940977 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:14.941402 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.942014 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.942896 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:14.943182 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:14.943338 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:14.946255 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:52069: BlockingRecv error: recv got EOF from 127.30.2.194:52069 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 14:46:15 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:15 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:15.144469 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:15.138398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:59631)
0923 14:46:15.138738 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:15.138751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:15.138884 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:15.139209 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:15.139212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:15.139427 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:15.139520 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:15.139525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.140385 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.140391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:15.140958 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:15.140965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.141465 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.141468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:15.141552 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:15.142158 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:15.142176 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:15.144361 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
W20260923 14:46:15.144796 7484 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:15.138515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:59631 (local address 127.30.2.254:41823)
0923 14:46:15.138683 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:15.138688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:15.138824 (+ 136us) server_negotiation.cc:407] Connection header received
0923 14:46:15.138982 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:15.138985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:15.139041 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:15.139169 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:15.139658 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.140275 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.141080 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.141337 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.141452 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:15.144524 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:59631: BlockingRecv error: recv got EOF from 127.30.2.195:59631 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 14:46:15 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:15 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:15.762655 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:15.757091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:48659)
0923 14:46:15.757434 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:15.757449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:15.757535 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:15.757814 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:15.757817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:15.758009 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:15.758152 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:15.758159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.758921 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.758924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:15.759582 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:15.759592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.760051 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.760054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:15.760153 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:15.760754 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:15.760766 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:15.762518 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":74,"threads_started":1}
W20260923 14:46:15.762836 7488 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:15.757171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:48659 (local address 127.30.2.254:41823)
0923 14:46:15.757470 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:15.757475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:15.757504 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:46:15.757644 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:15.757646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:15.757689 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:15.757761 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:15.758263 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.758804 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.759694 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.759939 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.760018 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:15.762699 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:48659: BlockingRecv error: recv got EOF from 127.30.2.193:48659 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
Sep 23 14:46:15 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:15 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:15.952996 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:15.947264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:46973)
0923 14:46:15.947551 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:15.947564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:15.947659 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:15.947911 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:15.947913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:15.948090 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:15.948187 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:15.948192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.948878 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.948881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:15.949431 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:15.949438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.949859 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.949863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:15.949949 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:15.950613 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:15.950657 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:15.952800 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":96,"threads_started":1}
W20260923 14:46:15.953106 7488 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:15.947317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:46973 (local address 127.30.2.254:41823)
0923 14:46:15.947466 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:15.947471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:15.947640 (+ 169us) server_negotiation.cc:407] Connection header received
0923 14:46:15.947740 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:15.947743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:15.947806 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:15.947873 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:15.948283 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.948789 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.949529 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:15.949738 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:15.949851 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:15.952951 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:46973: BlockingRecv error: recv got EOF from 127.30.2.194:46973 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 14:46:16 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:16 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:16.150990 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:16.145360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:58957)
0923 14:46:16.145681 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:16.145699 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:16.145822 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:16.146129 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:16.146132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:16.146364 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:16.146510 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:16.146521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.147432 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.147435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:16.148030 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:16.148040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.148500 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.148503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:16.148581 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:16.149066 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:16.149078 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:16.150867 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":120,"threads_started":1}
W20260923 14:46:16.151161 7488 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:16.145514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:58957 (local address 127.30.2.254:41823)
0923 14:46:16.145665 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:16.145669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:16.145798 (+ 129us) server_negotiation.cc:407] Connection header received
0923 14:46:16.145914 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:16.145918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:16.145986 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:16.146058 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:16.146628 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.147329 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.148147 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.148365 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.148452 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:16.151002 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:58957: BlockingRecv error: recv got EOF from 127.30.2.195:58957 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 14:46:16 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:16 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:16.769222 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:16.763533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:52225)
0923 14:46:16.763920 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:16.763934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:16.764041 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:16.764289 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:16.764291 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:16.764483 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:16.764605 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:16.764610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.765376 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.765379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:16.766008 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:16.766019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.766478 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.766482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:16.766550 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:16.767164 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:16.767177 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:16.769045 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":87,"threads_started":1}
W20260923 14:46:16.769318 7492 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:16.763681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:52225 (local address 127.30.2.254:41823)
0923 14:46:16.763933 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:16.763937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:16.764020 (+ 83us) server_negotiation.cc:407] Connection header received
0923 14:46:16.764142 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:16.764145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:16.764194 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:16.764273 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:16.764722 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.765265 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.766126 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.766354 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.766437 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:16.769186 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:52225: BlockingRecv error: recv got EOF from 127.30.2.193:52225 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":58,"threads_started":1}
Sep 23 14:46:16 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:16 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:16.960050 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:16.953883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:36451)
0923 14:46:16.954174 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:16.954200 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:16.954308 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:16.954653 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:16.954655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:16.954846 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:16.954952 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:16.954958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.955765 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.955768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:16.956471 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:16.956482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.956921 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.956925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:16.956993 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:16.957526 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:16.957548 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:16.959877 (+ 2329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
W20260923 14:46:16.960098 7492 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:16.953995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:36451 (local address 127.30.2.254:41823)
0923 14:46:16.954147 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:16.954151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:16.954274 (+ 123us) server_negotiation.cc:407] Connection header received
0923 14:46:16.954487 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:16.954489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:16.954534 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:16.954610 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:16.955060 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.955649 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.956593 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:16.956814 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:16.956894 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:16.959997 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:36451: BlockingRecv error: recv got EOF from 127.30.2.194:36451 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 14:46:17 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:17 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:17.157502 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:17.151831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:37323)
0923 14:46:17.152155 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:17.152174 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:17.152276 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:17.152592 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:17.152595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:17.152791 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:17.152892 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:17.152897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.153663 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.153666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:17.154230 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:17.154238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.154672 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.154675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:17.154737 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:17.155328 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:17.155341 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:17.157363 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
W20260923 14:46:17.157642 7492 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:17.152010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:37323 (local address 127.30.2.254:41823)
0923 14:46:17.152206 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:17.152210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:17.152225 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:46:17.152400 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:17.152403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:17.152454 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:17.152528 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:17.153005 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.153549 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.154343 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.154563 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.154646 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:17.157525 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:37323: BlockingRecv error: recv got EOF from 127.30.2.195:37323 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 23 14:46:17 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:17 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:17.776106 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:17.770101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:35347)
0923 14:46:17.770393 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:17.770407 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:17.770494 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:17.770760 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:17.770762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:17.770940 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:17.771039 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:17.771044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.771824 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.771827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:17.772393 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:17.772401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.772959 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.772963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:17.773089 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:17.773673 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:17.773688 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:17.775958 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":113,"threads_started":1}
W20260923 14:46:17.776296 7496 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:17.770206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:35347 (local address 127.30.2.254:41823)
0923 14:46:17.770442 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:17.770445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:17.770458 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:46:17.770595 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:17.770597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:17.770641 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:17.770719 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:17.771144 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.771688 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.772561 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.772808 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.772897 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:17.776096 (+ 3199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:35347: BlockingRecv error: recv got EOF from 127.30.2.193:35347 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
Sep 23 14:46:17 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:17 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:17.967057 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:17.961070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:34857)
0923 14:46:17.961398 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:17.961412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:17.961528 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:17.961794 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:17.961796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:17.961984 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:17.962101 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:17.962107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.962873 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.962876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:17.963457 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:17.963466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.964136 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.964140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:17.964232 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:17.964935 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:17.964953 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:17.966934 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":83,"threads_started":1}
W20260923 14:46:17.967211 7496 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:17.961171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:34857 (local address 127.30.2.254:41823)
0923 14:46:17.961373 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:17.961377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:17.961476 (+ 99us) server_negotiation.cc:407] Connection header received
0923 14:46:17.961632 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:17.961635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:17.961678 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:17.961745 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:17.962219 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.962763 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.963593 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:17.963982 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:17.964101 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:17.967078 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:34857: BlockingRecv error: recv got EOF from 127.30.2.194:34857 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 23 14:46:18 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:18 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:18.170181 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:18.158381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:46907)
0923 14:46:18.158636 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:18.158650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:18.158749 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:18.158960 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:18.158962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:18.159140 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:18.159234 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:18.159239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.160341 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.160344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:18.161040 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:18.161050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.161465 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.161468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:18.161525 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:18.168124 (+ 6599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:18.168139 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:18.170040 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
W20260923 14:46:18.170354 7496 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:18.158481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:46907 (local address 127.30.2.254:41823)
0923 14:46:18.158663 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:18.158666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:18.158723 (+ 57us) server_negotiation.cc:407] Connection header received
0923 14:46:18.158832 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:18.158835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:18.158882 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:18.158949 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:18.159361 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.160238 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.161149 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.161362 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.161440 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:18.170227 (+ 8787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:46907: BlockingRecv error: recv got EOF from 127.30.2.195:46907 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 23 14:46:18 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:18 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:18.782963 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:18.776915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:46941)
0923 14:46:18.777190 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:18.777213 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:18.777306 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:18.777551 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:18.777553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:18.777755 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:18.777855 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:18.777861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.778715 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.778718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:18.779292 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:18.779302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.779724 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.779727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:18.779821 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:18.780499 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:18.780514 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:18.782772 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
W20260923 14:46:18.783033 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:18.777021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:46941 (local address 127.30.2.254:41823)
0923 14:46:18.777282 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:18.777286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:18.777299 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:46:18.777387 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:18.777389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:18.777432 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:18.777515 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:18.778010 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.778566 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.779404 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.779623 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.779699 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:18.782903 (+ 3204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:46941: BlockingRecv error: recv got EOF from 127.30.2.193:46941 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":112,"threads_started":1}
Sep 23 14:46:18 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:18 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:18.975389 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:18.968015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:44775)
0923 14:46:18.968309 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:18.968326 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:18.968483 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:18.968784 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:18.968787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:18.969047 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:18.969180 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:18.969189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.970481 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.970487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:18.971521 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:18.971537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.972227 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.972233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:18.972358 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:18.973026 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:18.973042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:18.975228 (+ 2186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":102,"threads_started":1}
W20260923 14:46:18.975683 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:18.968197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:44775 (local address 127.30.2.254:41823)
0923 14:46:18.968329 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:18.968333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:18.968500 (+ 167us) server_negotiation.cc:407] Connection header received
0923 14:46:18.968599 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:18.968602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:18.968662 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:18.968736 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:18.969310 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.970318 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.971679 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:18.972099 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:18.972223 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:18.975380 (+ 3157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:44775: BlockingRecv error: recv got EOF from 127.30.2.194:44775 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 23 14:46:19 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:19 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:19.176878 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:19.170988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:42877)
0923 14:46:19.171285 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:19.171298 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:19.171392 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:19.171617 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:19.171619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:19.171822 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:19.171914 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:19.171920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.172683 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.172686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:19.173238 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:19.173255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.173754 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.173758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:19.173831 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:19.174428 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:19.174441 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:19.176742 (+ 2301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":118,"threads_started":1}
W20260923 14:46:19.177030 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:19.171071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:42877 (local address 127.30.2.254:41823)
0923 14:46:19.171229 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:19.171233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:19.171365 (+ 132us) server_negotiation.cc:407] Connection header received
0923 14:46:19.171468 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:19.171471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:19.171517 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:19.171582 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:19.172037 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.172583 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.173368 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.173625 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.173705 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:19.176877 (+ 3172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:42877: BlockingRecv error: recv got EOF from 127.30.2.195:42877 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 23 14:46:19 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:19 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:19.789415 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:19.783847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:46937)
0923 14:46:19.784205 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:19.784219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:19.784312 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:19.784550 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:19.784553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:19.784735 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:19.784831 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:19.784836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.785691 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.785695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:19.786253 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:19.786261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.786750 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.786753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:19.786827 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:19.787348 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:19.787359 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:19.789237 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":95,"threads_started":1}
W20260923 14:46:19.789661 7504 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:19.783996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:46937 (local address 127.30.2.254:41823)
0923 14:46:19.784313 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:19.784317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:19.784332 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:46:19.784373 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:19.784375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:19.784419 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:19.784507 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:19.784943 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.785564 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.786387 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.786643 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.786736 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:19.789513 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:46937: BlockingRecv error: recv got EOF from 127.30.2.193:46937 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":66,"threads_started":1}
Sep 23 14:46:19 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:19 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:19.981886 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:19.976289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:52983)
0923 14:46:19.976567 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:19.976587 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:19.976727 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:19.976995 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:19.976999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:19.977253 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:19.977357 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:19.977364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.978152 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.978156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:19.978707 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:19.978715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.979179 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.979183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:19.979267 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:19.979844 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:19.979855 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:19.981719 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
W20260923 14:46:19.981942 7504 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:19.976355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:52983 (local address 127.30.2.254:41823)
0923 14:46:19.976533 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:19.976538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:19.976667 (+ 129us) server_negotiation.cc:407] Connection header received
0923 14:46:19.976827 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:19.976830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:19.976883 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:19.976972 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:19.977482 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.978031 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.978828 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:19.979064 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:19.979175 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:19.981837 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:52983: BlockingRecv error: recv got EOF from 127.30.2.194:52983 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 23 14:46:20 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:20 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.195@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:20.183607 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:20.177748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.195:35031)
0923 14:46:20.178155 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:20.178177 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:20.178316 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:20.178630 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:20.178633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:20.178847 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:20.178942 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:20.178948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.179852 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.179857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:20.180486 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:20.180496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.180968 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.180971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:20.181060 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:20.181579 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:20.181590 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:20.183427 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":95,"threads_started":1}
W20260923 14:46:20.183665 7504 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:20.177880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:35031 (local address 127.30.2.254:41823)
0923 14:46:20.178065 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:20.178070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:20.178257 (+ 187us) server_negotiation.cc:407] Connection header received
0923 14:46:20.178425 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:20.178430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:20.178485 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:20.178604 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:20.179071 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.179712 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.180619 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.180854 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.180964 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:20.183541 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.195:35031: BlockingRecv error: recv got EOF from 127.30.2.195:35031 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 23 14:46:20 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:20 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.193@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:20.796139 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:20.790420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.193:34201)
0923 14:46:20.790708 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:20.790721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:20.790814 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:20.791035 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:20.791037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:20.791227 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:20.791395 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:20.791406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.792303 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.792308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:20.792900 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:20.792910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.793392 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.793396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:20.793486 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:20.794001 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:20.794013 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:20.795961 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
W20260923 14:46:20.796216 7508 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:20.790468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:34201 (local address 127.30.2.254:41823)
0923 14:46:20.790721 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:20.790725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:20.790770 (+ 45us) server_negotiation.cc:407] Connection header received
0923 14:46:20.790881 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:20.790883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:20.790924 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:20.790998 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:20.791538 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.792167 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.793031 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.793266 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.793375 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:20.796091 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.193:34201: BlockingRecv error: recv got EOF from 127.30.2.193:34201 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
Sep 23 14:46:20 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:46:20 dist-test-slave-wjf2 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.2.194@KRBTEST.COM for oryx/127.30.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:46:20.988998 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:20.982727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:41823 (local address 127.30.2.194:53065)
0923 14:46:20.983063 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:20.983080 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:20.983241 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:20.983537 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:20.983540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:20.983767 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:20.983899 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:20.983906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.984767 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.984772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:20.985399 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:20.985407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.985993 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.985997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:20.986106 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:20.986758 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:20.986775 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:20.988867 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.2.254:41823: Server oryx/127.30.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":85,"threads_started":1}
W20260923 14:46:20.989079 7508 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:20.982860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:53065 (local address 127.30.2.254:41823)
0923 14:46:20.983038 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:20.983043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:20.983175 (+ 132us) server_negotiation.cc:407] Connection header received
0923 14:46:20.983349 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:20.983354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:20.983405 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:20.983510 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:20.984031 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.984640 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.985532 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:20.985837 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:20.985957 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:20.988989 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.2.194:53065: BlockingRecv error: recv got EOF from 127.30.2.194:53065 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 14:46:21.054939 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 7033
I20260923 14:46:21.061097 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 7185
I20260923 14:46:21.066514 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 7317
I20260923 14:46:21.071923 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 6963
2026-09-23T14:46:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18060 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T14:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:46:21Z Disabled control of system clock
I20260923 14:46:21.101994 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:39527
--webserver_interface=127.30.2.254
--webserver_port=0
--builtin_ntp_servers=127.30.2.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:39527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 14:46:21.209949 7513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:21.210213 7513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:21.210249 7513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:21.214257 7513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:46:21.214308 7513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:21.214323 7513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:21.214340 7513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:46:21.214355 7513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:46:21.219045 7513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:39527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:39527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:21.220155 7513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:21.221489 7513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:46:21.230525 7513 server_base.cc:1034] running on GCE node
W20260923 14:46:21.230496 7519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:21.230472 7521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:21.230450 7518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:21.231312 7513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:21.232475 7513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:21.233645 7513 hybrid_clock.cc:648] HybridClock initialized: now 1790174781233634 us; error 28 us; skew 500 ppm
I20260923 14:46:21.235502 7513 webserver.cc:460] Webserver started at http://127.30.2.254:35001/ using document root <none> and password file <none>
I20260923 14:46:21.236155 7513 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:21.236218 7513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:21.236434 7513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:21.238438 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "8cf5a81aea914a55afb63b18a200ee5c"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "177941afec9c94d002f7b9ffb4293b2f"
server_key_iv: "8ba2422b2a702ed94bfa18b7b84e6ba2"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.239007 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "8cf5a81aea914a55afb63b18a200ee5c"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "177941afec9c94d002f7b9ffb4293b2f"
server_key_iv: "8ba2422b2a702ed94bfa18b7b84e6ba2"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.242859 7513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 14:46:21.246138 7527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:21.247678 7513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 14:46:21.247874 7513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "8cf5a81aea914a55afb63b18a200ee5c"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "177941afec9c94d002f7b9ffb4293b2f"
server_key_iv: "8ba2422b2a702ed94bfa18b7b84e6ba2"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.248004 7513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:21.265710 7513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:21.266350 7513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:21.266536 7513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:21.273579 7513 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:39527
I20260923 14:46:21.273588 7579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:39527 every 8 connection(s)
I20260923 14:46:21.274489 7513 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:46:21.277392 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 7513
I20260923 14:46:21.277506 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:46:21.277861 30731 external_mini_cluster.cc:1428] Setting key 3d536b85c6b6befa28dd93d59e031105
I20260923 14:46:21.277967 7580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:21.285396 7580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c: Bootstrap starting.
I20260923 14:46:21.287823 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:21.279356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.30.2.254:39527)
0923 14:46:21.280082 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:21.280100 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:21.280139 (+ 39us) server_negotiation.cc:407] Connection header received
0923 14:46:21.280788 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:21.280805 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:21.281202 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:21.281486 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:21.281913 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.283110 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.284495 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.284948 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.285073 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:21.286034 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:21.286039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:21.286051 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:46:21.286435 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:21.286485 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:46:21.286504 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:21.286579 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:21.286769 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 14:46:21.287054 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":157,"threads_started":1}
I20260923 14:46:21.288165 7580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:21.289099 7580 log.cc:824] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c: Log is configured to *not* fsync() on all Append() calls
I20260923 14:46:21.291247 7580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c: No bootstrap required, opened a new log
I20260923 14:46:21.295095 7580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf5a81aea914a55afb63b18a200ee5c" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39527 } }
I20260923 14:46:21.295312 7580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:21.295388 7580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf5a81aea914a55afb63b18a200ee5c, State: Initialized, Role: FOLLOWER
I20260923 14:46:21.295904 7580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf5a81aea914a55afb63b18a200ee5c" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39527 } }
I20260923 14:46:21.296072 7580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:46:21.296145 7580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:46:21.296257 7580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:21.297151 7580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf5a81aea914a55afb63b18a200ee5c" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39527 } }
I20260923 14:46:21.297504 7580 leader_election.cc:302] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cf5a81aea914a55afb63b18a200ee5c; no voters:
I20260923 14:46:21.297838 7580 leader_election.cc:288] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:46:21.297950 7585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:46:21.298200 7585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [term 1 LEADER]: Becoming Leader. State: Replica: 8cf5a81aea914a55afb63b18a200ee5c, State: Running, Role: LEADER
I20260923 14:46:21.298527 7585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf5a81aea914a55afb63b18a200ee5c" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39527 } }
I20260923 14:46:21.299115 7580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:46:21.300209 7586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cf5a81aea914a55afb63b18a200ee5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf5a81aea914a55afb63b18a200ee5c" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39527 } } }
I20260923 14:46:21.300269 7587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cf5a81aea914a55afb63b18a200ee5c. Latest consensus state: current_term: 1 leader_uuid: "8cf5a81aea914a55afb63b18a200ee5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf5a81aea914a55afb63b18a200ee5c" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39527 } } }
I20260923 14:46:21.300410 7586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [sys.catalog]: This master's current role is: LEADER
I20260923 14:46:21.300414 7587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c [sys.catalog]: This master's current role is: LEADER
I20260923 14:46:21.300885 7594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:46:21.304052 7594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:46:21.310010 7594 catalog_manager.cc:1347] Generated new cluster ID: f89d043a3abf450287c5c0e12adbc8e9
I20260923 14:46:21.310082 7594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:46:21.320492 7594 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:46:21.321659 7594 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:46:21.327293 7594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cf5a81aea914a55afb63b18a200ee5c: Generated new TSK 0
I20260923 14:46:21.328045 7594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:46:21.332676 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39527
--builtin_ntp_servers=127.30.2.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 14:46:21.444882 7604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:21.445124 7604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:21.445161 7604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:21.449138 7604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:21.449186 7604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:21.449239 7604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:46:21.453881 7604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:21.454886 7604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:21.456225 7604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:21.465942 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:21.465978 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:21.466022 7604 server_base.cc:1034] running on GCE node
W20260923 14:46:21.465942 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:21.466612 7604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:21.467257 7604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:21.468454 7604 hybrid_clock.cc:648] HybridClock initialized: now 1790174781468426 us; error 42 us; skew 500 ppm
I20260923 14:46:21.470352 7604 webserver.cc:460] Webserver started at http://127.30.2.193:43123/ using document root <none> and password file <none>
I20260923 14:46:21.470944 7604 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:21.471035 7604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:21.471269 7604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:21.473320 7604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "85765c6a0db342bd8157d0542c9d70d8"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "f5d02378119a63f0d962293fec9f2a20"
server_key_iv: "c8c4bc5ae2e37cc0bc93d450b373e658"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.473870 7604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "85765c6a0db342bd8157d0542c9d70d8"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "f5d02378119a63f0d962293fec9f2a20"
server_key_iv: "c8c4bc5ae2e37cc0bc93d450b373e658"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.477500 7604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260923 14:46:21.480052 7618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:21.481338 7604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:46:21.481473 7604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "85765c6a0db342bd8157d0542c9d70d8"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "f5d02378119a63f0d962293fec9f2a20"
server_key_iv: "c8c4bc5ae2e37cc0bc93d450b373e658"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.481586 7604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:21.504298 7604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:21.504971 7604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:21.505172 7604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:21.505757 7604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:21.506690 7604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:21.506765 7604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:21.506855 7604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:21.506906 7604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:21.515610 7604 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:33231
I20260923 14:46:21.515640 7731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:33231 every 8 connection(s)
I20260923 14:46:21.516538 7604 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:46:21.518987 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 7604
I20260923 14:46:21.519111 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:46:21.519410 30731 external_mini_cluster.cc:1428] Setting key dffa09523bb049daf3480315c6b5000a
I20260923 14:46:21.521421 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39527
--builtin_ntp_servers=127.30.2.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 14:46:21.526165 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:21.518385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:53543 (local address 127.30.2.254:39527)
0923 14:46:21.518548 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:21.518553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:21.519720 (+ 1167us) server_negotiation.cc:407] Connection header received
0923 14:46:21.521293 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:21.521296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:21.521351 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:21.521445 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:21.522411 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.523025 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.524221 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.524454 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.524577 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:21.525685 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:21.525689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:21.525691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:46:21.525702 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:21.525720 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:46:21.525727 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:21.525801 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:21.525968 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 14:46:21.526026 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:46:21.526986 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:21.518802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:39527 (local address 127.30.2.193:53543)
0923 14:46:21.519543 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:21.519594 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:21.521085 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:21.521657 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:21.521694 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:21.521786 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:21.522267 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:21.522279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.523162 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.523165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:21.524092 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:21.524102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.524613 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.524618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:21.524784 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:21.525435 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:46:21.525436 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:21.525474 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:46:21.525532 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:21.525818 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:21.525834 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:21.525839 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:46:21.525956 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 14:46:21.526234 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":134,"threads_started":1}
I20260923 14:46:21.528312 7732 heartbeater.cc:347] Connected to a master server at 127.30.2.254:39527
I20260923 14:46:21.528537 7732 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:21.529029 7732 heartbeater.cc:511] Master 127.30.2.254:39527 requested a full tablet report, sending...
I20260923 14:46:21.530534 7544 ts_manager.cc:194] Registered new tserver with Master: 85765c6a0db342bd8157d0542c9d70d8 (127.30.2.193:33231)
I20260923 14:46:21.531654 7544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.193:53543
W20260923 14:46:21.632615 7735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:21.632916 7735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:21.632988 7735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:21.637174 7735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:21.637251 7735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:21.637344 7735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:46:21.642066 7735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:21.643251 7735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:21.644826 7735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:21.655011 7743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:21.655017 7740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:21.655017 7741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:21.655323 7735 server_base.cc:1034] running on GCE node
I20260923 14:46:21.655874 7735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:21.656512 7735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:21.657732 7735 hybrid_clock.cc:648] HybridClock initialized: now 1790174781657702 us; error 38 us; skew 500 ppm
I20260923 14:46:21.659763 7735 webserver.cc:460] Webserver started at http://127.30.2.194:43233/ using document root <none> and password file <none>
I20260923 14:46:21.660435 7735 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:21.660501 7735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:21.660774 7735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:21.662884 7735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "780d73b09c064b9a9c4799c1d5090388"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "22713456aa77baa570b00c8fb9be931e"
server_key_iv: "af1cdaaa06bf82613d6c5d49c60f4832"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.663486 7735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "780d73b09c064b9a9c4799c1d5090388"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "22713456aa77baa570b00c8fb9be931e"
server_key_iv: "af1cdaaa06bf82613d6c5d49c60f4832"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.667524 7735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260923 14:46:21.670171 7749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:21.671471 7735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:46:21.671618 7735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "780d73b09c064b9a9c4799c1d5090388"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "22713456aa77baa570b00c8fb9be931e"
server_key_iv: "af1cdaaa06bf82613d6c5d49c60f4832"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.671738 7735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:21.689126 7735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:21.689813 7735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:21.690006 7735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:21.690595 7735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:21.691522 7735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:21.691609 7735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:21.691704 7735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:21.691756 7735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:21.700769 7735 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:39299
I20260923 14:46:21.700832 7862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:39299 every 8 connection(s)
I20260923 14:46:21.701758 7735 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:46:21.707574 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 7735
I20260923 14:46:21.707705 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:46:21.708017 30731 external_mini_cluster.cc:1428] Setting key 085b1e7c805d908f5a9a26a59394b934
I20260923 14:46:21.710391 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39527
--builtin_ntp_servers=127.30.2.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 14:46:21.711179 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:21.703564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:54645 (local address 127.30.2.254:39527)
0923 14:46:21.703710 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:21.703715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:21.704723 (+ 1008us) server_negotiation.cc:407] Connection header received
0923 14:46:21.706133 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:21.706137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:21.706198 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:21.706305 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:21.707174 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.707771 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.709029 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.709277 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.709394 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:21.710625 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:21.710632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:21.710638 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:46:21.710654 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:21.710678 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:46:21.710691 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:21.710739 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:21.710995 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 14:46:21.711064 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:46:21.712255 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:21.703914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:39527 (local address 127.30.2.194:54645)
0923 14:46:21.704552 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:21.704592 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:21.705882 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:21.706472 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:21.706479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:21.706557 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:21.707022 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:21.707035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.707904 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.707912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:21.708901 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:21.708910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.709428 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.709432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:21.709590 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:21.710293 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:46:21.710298 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:21.710353 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:46:21.710434 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:21.710811 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:21.710822 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:21.710831 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 14:46:21.710959 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 14:46:21.711256 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260923 14:46:21.713375 7863 heartbeater.cc:347] Connected to a master server at 127.30.2.254:39527
I20260923 14:46:21.713577 7863 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:21.714020 7863 heartbeater.cc:511] Master 127.30.2.254:39527 requested a full tablet report, sending...
I20260923 14:46:21.715042 7544 ts_manager.cc:194] Registered new tserver with Master: 780d73b09c064b9a9c4799c1d5090388 (127.30.2.194:39299)
I20260923 14:46:21.715616 7544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.194:54645
W20260923 14:46:21.822399 7866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:21.822685 7866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:21.822798 7866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:21.829814 7866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:21.829895 7866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:21.830015 7866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:46:21.835906 7866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:21.837002 7866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:21.838366 7866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:21.847749 7872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:21.847738 7874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:21.847772 7871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:21.848019 7866 server_base.cc:1034] running on GCE node
I20260923 14:46:21.848426 7866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:21.849077 7866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:21.850241 7866 hybrid_clock.cc:648] HybridClock initialized: now 1790174781850221 us; error 27 us; skew 500 ppm
I20260923 14:46:21.852205 7866 webserver.cc:460] Webserver started at http://127.30.2.195:46877/ using document root <none> and password file <none>
I20260923 14:46:21.852799 7866 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:21.852861 7866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:21.853114 7866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:21.855180 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "139220601839422fb7c0dff59509a4f9"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "f4d64bcbba73f862263d19471efbd5cf"
server_key_iv: "0c8b1df28d6bc210bb7bab31f403405e"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.855751 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "139220601839422fb7c0dff59509a4f9"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "f4d64bcbba73f862263d19471efbd5cf"
server_key_iv: "0c8b1df28d6bc210bb7bab31f403405e"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.859388 7866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:46:21.862022 7880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:21.863305 7866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:46:21.863425 7866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "139220601839422fb7c0dff59509a4f9"
format_stamp: "Formatted at 2026-09-23 14:46:21 on dist-test-slave-wjf2"
server_key: "f4d64bcbba73f862263d19471efbd5cf"
server_key_iv: "0c8b1df28d6bc210bb7bab31f403405e"
server_key_version: "encryptionkey@0"
I20260923 14:46:21.863529 7866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:21.876866 7866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:21.877530 7866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:21.877718 7866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:21.878294 7866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:21.879196 7866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:21.879251 7866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:21.879316 7866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:21.879352 7866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:21.888731 7866 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:42027
I20260923 14:46:21.888748 7993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:42027 every 8 connection(s)
I20260923 14:46:21.889773 7866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:46:21.896914 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 7866
I20260923 14:46:21.897010 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:46:21.897303 30731 external_mini_cluster.cc:1428] Setting key defc61e19059d2480c17336d34d1ffe5
I20260923 14:46:21.899914 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:21.891749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:55531 (local address 127.30.2.254:39527)
0923 14:46:21.891969 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:21.891976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:21.893140 (+ 1164us) server_negotiation.cc:407] Connection header received
0923 14:46:21.894726 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:21.894729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:21.894780 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:21.894868 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:21.895770 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.896408 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.897680 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.897993 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.898115 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:21.899348 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:21.899352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:21.899355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:46:21.899367 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:21.899389 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:46:21.899400 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:21.899477 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:21.899666 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 14:46:21.899735 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 14:46:21.900892 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:21.892289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:39527 (local address 127.30.2.195:55531)
0923 14:46:21.892994 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:21.893034 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:21.894506 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:21.895035 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:21.895042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:21.895155 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:21.895643 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:21.895655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.896519 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.896521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:21.897542 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:21.897552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:21.898182 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:21.898188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:21.898385 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:21.899053 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:46:21.899054 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:21.899097 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:46:21.899170 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:21.899500 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:21.899509 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:21.899514 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:46:21.899652 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 14:46:21.899967 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
I20260923 14:46:21.901952 7994 heartbeater.cc:347] Connected to a master server at 127.30.2.254:39527
I20260923 14:46:21.902222 7994 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:21.902755 7994 heartbeater.cc:511] Master 127.30.2.254:39527 requested a full tablet report, sending...
I20260923 14:46:21.903888 7544 ts_manager.cc:194] Registered new tserver with Master: 139220601839422fb7c0dff59509a4f9 (127.30.2.195:42027)
I20260923 14:46:21.904500 7544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.195:55531
I20260923 14:46:21.911863 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 14:46:21.914992 7583 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:21.913938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.30.2.254:39527)
0923 14:46:21.914124 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:21.914129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:21.914278 (+ 149us) server_negotiation.cc:407] Connection header received
0923 14:46:21.914453 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:21.914460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:21.914503 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:21.914621 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:21.914883 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44252: BlockingRecv error: recv got EOF from 127.0.0.1:44252 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 14:46:21.915541 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 7604
I20260923 14:46:21.921661 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 7735
I20260923 14:46:21.927085 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 7866
I20260923 14:46:21.932786 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 7513
2026-09-23T14:46:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (859 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T14:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:46:21Z Disabled control of system clock
I20260923 14:46:21.962437 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44773
--webserver_interface=127.30.2.254
--webserver_port=0
--builtin_ntp_servers=127.30.2.212:37365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:44773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 14:46:22.072518 8005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:22.072790 8005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:22.072829 8005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:22.076836 8005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:46:22.076889 8005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:22.076905 8005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:22.076920 8005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:46:22.076936 8005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:46:22.081581 8005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:44773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:22.082590 8005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:22.084007 8005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:22.092229 8010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:22.092247 8011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:22.092316 8013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:22.092425 8005 server_base.cc:1034] running on GCE node
I20260923 14:46:22.092994 8005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:22.093976 8005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:22.095140 8005 hybrid_clock.cc:648] HybridClock initialized: now 1790174782095120 us; error 34 us; skew 500 ppm
I20260923 14:46:22.096900 8005 webserver.cc:460] Webserver started at http://127.30.2.254:44491/ using document root <none> and password file <none>
I20260923 14:46:22.097509 8005 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:22.097571 8005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:22.097743 8005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:22.099686 8005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "54c00f4d1da14001b577775ac6e69c21"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "b9dacc422b9efc2af54b1d43bff08a9d"
server_key_iv: "6304c25752b9e3cb8d4d8cdc07a76ec5"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.100248 8005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "54c00f4d1da14001b577775ac6e69c21"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "b9dacc422b9efc2af54b1d43bff08a9d"
server_key_iv: "6304c25752b9e3cb8d4d8cdc07a76ec5"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.103703 8005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:46:22.106070 8019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:22.107404 8005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:46:22.107545 8005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "54c00f4d1da14001b577775ac6e69c21"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "b9dacc422b9efc2af54b1d43bff08a9d"
server_key_iv: "6304c25752b9e3cb8d4d8cdc07a76ec5"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.107661 8005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:22.112064 8005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:22.112650 8005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:22.112833 8005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:22.119732 8005 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:44773
I20260923 14:46:22.119742 8071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:44773 every 8 connection(s)
I20260923 14:46:22.120747 8005 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:46:22.124003 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:22.127679 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 8005
I20260923 14:46:22.127835 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:46:22.128105 30731 external_mini_cluster.cc:1428] Setting key 93f0e66801b4d600df61376995daa0b7
I20260923 14:46:22.129956 8072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21: Bootstrap starting.
I20260923 14:46:22.132697 8072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:22.133582 8072 log.cc:824] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21: Log is configured to *not* fsync() on all Append() calls
I20260923 14:46:22.134151 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:22.129560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.30.2.254:44773)
0923 14:46:22.130229 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:22.130250 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:22.130292 (+ 42us) server_negotiation.cc:407] Connection header received
0923 14:46:22.131132 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:22.131163 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:22.131616 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:22.131998 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:22.132693 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:22.132699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:22.132715 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:46:22.132727 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:22.132797 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:46:22.132824 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:22.132962 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:22.133034 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 14:46:22.133265 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
I20260923 14:46:22.135532 8072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21: No bootstrap required, opened a new log
I20260923 14:46:22.139295 8072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c00f4d1da14001b577775ac6e69c21" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44773 } }
I20260923 14:46:22.139498 8072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:22.139576 8072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c00f4d1da14001b577775ac6e69c21, State: Initialized, Role: FOLLOWER
I20260923 14:46:22.140033 8072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c00f4d1da14001b577775ac6e69c21" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44773 } }
I20260923 14:46:22.140194 8072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:46:22.140257 8072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:46:22.140358 8072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:22.141182 8072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c00f4d1da14001b577775ac6e69c21" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44773 } }
I20260923 14:46:22.141512 8072 leader_election.cc:302] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54c00f4d1da14001b577775ac6e69c21; no voters:
I20260923 14:46:22.141801 8072 leader_election.cc:288] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:46:22.141921 8077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:46:22.142164 8077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [term 1 LEADER]: Becoming Leader. State: Replica: 54c00f4d1da14001b577775ac6e69c21, State: Running, Role: LEADER
I20260923 14:46:22.142498 8077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c00f4d1da14001b577775ac6e69c21" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44773 } }
I20260923 14:46:22.142935 8072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:46:22.144032 8079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54c00f4d1da14001b577775ac6e69c21. Latest consensus state: current_term: 1 leader_uuid: "54c00f4d1da14001b577775ac6e69c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c00f4d1da14001b577775ac6e69c21" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44773 } } }
I20260923 14:46:22.144156 8079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [sys.catalog]: This master's current role is: LEADER
I20260923 14:46:22.144385 8078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54c00f4d1da14001b577775ac6e69c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c00f4d1da14001b577775ac6e69c21" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44773 } } }
I20260923 14:46:22.144527 8078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21 [sys.catalog]: This master's current role is: LEADER
I20260923 14:46:22.144918 8086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:46:22.147562 8086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:46:22.152776 8086 catalog_manager.cc:1347] Generated new cluster ID: 9cbff1bae0cc4966bce61547f3da20ae
I20260923 14:46:22.152840 8086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:46:22.178105 8086 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:46:22.178431 8086 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:46:22.184602 8086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54c00f4d1da14001b577775ac6e69c21: Generated new TSK 0
I20260923 14:46:22.185257 8086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:46:22.194746 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44773
--builtin_ntp_servers=127.30.2.212:37365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 14:46:22.311260 8096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:22.311676 8096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:22.311746 8096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:22.315744 8096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:22.315819 8096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:22.315876 8096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:46:22.320744 8096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:22.321906 8096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:22.323354 8096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:22.333218 8101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:22.333246 8104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:22.333357 8102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:22.333793 8096 server_base.cc:1034] running on GCE node
I20260923 14:46:22.334261 8096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:22.334900 8096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:22.336105 8096 hybrid_clock.cc:648] HybridClock initialized: now 1790174782336067 us; error 52 us; skew 500 ppm
I20260923 14:46:22.338116 8096 webserver.cc:460] Webserver started at http://127.30.2.193:44583/ using document root <none> and password file <none>
I20260923 14:46:22.338796 8096 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:22.338925 8096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:22.339170 8096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:22.341289 8096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "42b971e485ea4f329c255cd5cd1afff1"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "ab19dbb2d10af3f8599cd6d7464c8576"
server_key_iv: "a8d421d45b015e9d4777b8a17c9d1877"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.341849 8096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "42b971e485ea4f329c255cd5cd1afff1"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "ab19dbb2d10af3f8599cd6d7464c8576"
server_key_iv: "a8d421d45b015e9d4777b8a17c9d1877"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.345844 8096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:46:22.348392 8110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:22.349586 8096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:46:22.349728 8096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "42b971e485ea4f329c255cd5cd1afff1"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "ab19dbb2d10af3f8599cd6d7464c8576"
server_key_iv: "a8d421d45b015e9d4777b8a17c9d1877"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.349843 8096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:22.354346 8096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:22.354951 8096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:22.355131 8096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:22.355666 8096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:22.356575 8096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:22.356647 8096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:22.356715 8096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:22.356765 8096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:22.365494 8096 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:40773
I20260923 14:46:22.365518 8223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:40773 every 8 connection(s)
I20260923 14:46:22.366336 8096 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:46:22.371529 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 8096
I20260923 14:46:22.371665 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:46:22.372020 30731 external_mini_cluster.cc:1428] Setting key 8133f198fb20d9d273b6fcfd6c66af5c
I20260923 14:46:22.372280 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:22.368131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:60727 (local address 127.30.2.254:44773)
0923 14:46:22.368329 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:22.368334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:22.369330 (+ 996us) server_negotiation.cc:407] Connection header received
0923 14:46:22.370546 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:22.370548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:22.370585 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:22.370658 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:22.371711 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:22.371719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:22.371722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:46:22.371735 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:22.371775 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:46:22.371785 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:22.371917 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:22.372082 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 14:46:22.372128 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 14:46:22.373395 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:22.368557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:44773 (local address 127.30.2.193:60727)
0923 14:46:22.369193 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:22.369231 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:22.370366 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:22.370783 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:22.370790 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:22.370871 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:22.371460 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:46:22.371461 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:22.371500 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:46:22.371563 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:22.371914 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:22.371924 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:22.371932 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 14:46:22.372108 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 14:46:22.372337 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260923 14:46:22.374037 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44773
--builtin_ntp_servers=127.30.2.212:37365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 14:46:22.374598 8224 heartbeater.cc:347] Connected to a master server at 127.30.2.254:44773
I20260923 14:46:22.374835 8224 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:22.375288 8224 heartbeater.cc:511] Master 127.30.2.254:44773 requested a full tablet report, sending...
I20260923 14:46:22.376933 8036 ts_manager.cc:194] Registered new tserver with Master: 42b971e485ea4f329c255cd5cd1afff1 (127.30.2.193:40773)
W20260923 14:46:22.484056 8227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:22.484347 8227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:22.484426 8227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:22.488432 8227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:22.488510 8227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:22.488597 8227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:46:22.493363 8227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:22.494427 8227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:22.495858 8227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:22.505355 8235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:22.505359 8233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:22.505379 8232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:22.505798 8227 server_base.cc:1034] running on GCE node
I20260923 14:46:22.506165 8227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:22.506884 8227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:22.508038 8227 hybrid_clock.cc:648] HybridClock initialized: now 1790174782508019 us; error 31 us; skew 500 ppm
I20260923 14:46:22.509915 8227 webserver.cc:460] Webserver started at http://127.30.2.194:45851/ using document root <none> and password file <none>
I20260923 14:46:22.510490 8227 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:22.510550 8227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:22.510766 8227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:22.512979 8227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "f09ad0397d8146c192534634fbb79f28"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "016348ee157b8a2a403f9ec458bfbb9d"
server_key_iv: "68c938c66da3cf2653244d0d52073d9f"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.513548 8227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "f09ad0397d8146c192534634fbb79f28"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "016348ee157b8a2a403f9ec458bfbb9d"
server_key_iv: "68c938c66da3cf2653244d0d52073d9f"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.517426 8227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:46:22.520120 8241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:22.521348 8227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:46:22.521443 8227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "f09ad0397d8146c192534634fbb79f28"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "016348ee157b8a2a403f9ec458bfbb9d"
server_key_iv: "68c938c66da3cf2653244d0d52073d9f"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.521575 8227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:22.526515 8227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:22.527120 8227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:22.527304 8227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:22.527896 8227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:22.528779 8227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:22.528825 8227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:22.528891 8227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:22.528924 8227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:22.537658 8227 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:38337
I20260923 14:46:22.537678 8354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:38337 every 8 connection(s)
I20260923 14:46:22.538537 8227 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:46:22.540352 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 8227
I20260923 14:46:22.540477 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:46:22.540719 30731 external_mini_cluster.cc:1428] Setting key 2b4962c43f51a0006a15b4ee729591b7
I20260923 14:46:22.542919 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44773
--builtin_ntp_servers=127.30.2.212:37365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 14:46:22.545383 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:22.540358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:49363 (local address 127.30.2.254:44773)
0923 14:46:22.540508 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:22.540512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:22.541700 (+ 1188us) server_negotiation.cc:407] Connection header received
0923 14:46:22.543403 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:22.543406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:22.543441 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:22.543502 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:22.544867 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:22.544872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:22.544877 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:46:22.544893 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:22.544937 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:46:22.544948 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:22.545040 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:22.545168 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 14:46:22.545217 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:46:22.546162 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:22.540778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:44773 (local address 127.30.2.194:49363)
0923 14:46:22.541559 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:22.541597 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:22.543169 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:22.543734 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:22.543746 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:22.543902 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:22.544547 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:46:22.544550 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:22.544604 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:46:22.544695 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:22.545036 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:22.545043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:22.545047 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 14:46:22.545173 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 14:46:22.545358 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260923 14:46:22.547407 8355 heartbeater.cc:347] Connected to a master server at 127.30.2.254:44773
I20260923 14:46:22.547634 8355 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:22.548074 8355 heartbeater.cc:511] Master 127.30.2.254:44773 requested a full tablet report, sending...
I20260923 14:46:22.549188 8036 ts_manager.cc:194] Registered new tserver with Master: f09ad0397d8146c192534634fbb79f28 (127.30.2.194:38337)
W20260923 14:46:22.652806 8358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:22.653041 8358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:22.653079 8358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:22.657045 8358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:22.657092 8358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:22.657150 8358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:46:22.661751 8358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:22.662683 8358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:22.664053 8358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:22.673722 8364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:22.673861 8358 server_base.cc:1034] running on GCE node
W20260923 14:46:22.673774 8363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:22.673755 8366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:22.674352 8358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:22.674988 8358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:22.676153 8358 hybrid_clock.cc:648] HybridClock initialized: now 1790174782676126 us; error 43 us; skew 500 ppm
I20260923 14:46:22.677940 8358 webserver.cc:460] Webserver started at http://127.30.2.195:35025/ using document root <none> and password file <none>
I20260923 14:46:22.678514 8358 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:22.678574 8358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:22.678849 8358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:22.680888 8358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "f5baafec99e142a3b9d2fcdf0795a23d"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "2f33a94b160874f9e094b986363c8931"
server_key_iv: "e439f342da2e64a2d270bc407e38e379"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.681434 8358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "f5baafec99e142a3b9d2fcdf0795a23d"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "2f33a94b160874f9e094b986363c8931"
server_key_iv: "e439f342da2e64a2d270bc407e38e379"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.685031 8358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:46:22.687419 8372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:22.688652 8358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:46:22.688784 8358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "f5baafec99e142a3b9d2fcdf0795a23d"
format_stamp: "Formatted at 2026-09-23 14:46:22 on dist-test-slave-wjf2"
server_key: "2f33a94b160874f9e094b986363c8931"
server_key_iv: "e439f342da2e64a2d270bc407e38e379"
server_key_version: "encryptionkey@0"
I20260923 14:46:22.688894 8358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:22.693250 8358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:22.693825 8358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:22.694000 8358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:22.694510 8358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:22.695461 8358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:22.695535 8358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:22.695596 8358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:22.695647 8358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:22.704358 8358 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:41085
I20260923 14:46:22.704377 8485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:41085 every 8 connection(s)
I20260923 14:46:22.705205 8358 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:46:22.709486 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 8358
I20260923 14:46:22.709604 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:46:22.709929 30731 external_mini_cluster.cc:1428] Setting key 051983613c225ed3cabe93ac1c16a31b
I20260923 14:46:22.711411 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:22.706994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:54135 (local address 127.30.2.254:44773)
0923 14:46:22.707137 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:22.707140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:22.708211 (+ 1071us) server_negotiation.cc:407] Connection header received
0923 14:46:22.709521 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:22.709526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:22.709577 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:22.709686 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:22.710917 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:22.710921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:22.710924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:46:22.710936 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:22.710964 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:46:22.710992 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:22.711077 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:22.711228 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 14:46:22.711278 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:46:22.712410 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:22.707391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:44773 (local address 127.30.2.195:54135)
0923 14:46:22.708066 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:22.708104 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:22.709289 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:22.709854 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:22.709861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:22.709946 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:22.710653 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:46:22.710655 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:22.710714 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:46:22.710782 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:22.711074 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:22.711080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:22.711085 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:46:22.711254 (+ 169us) client_negotiation.cc:240] Negotiation successful
0923 14:46:22.711504 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260923 14:46:22.713481 8486 heartbeater.cc:347] Connected to a master server at 127.30.2.254:44773
I20260923 14:46:22.713713 8486 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:22.714107 8486 heartbeater.cc:511] Master 127.30.2.254:44773 requested a full tablet report, sending...
I20260923 14:46:22.715106 8036 ts_manager.cc:194] Registered new tserver with Master: f5baafec99e142a3b9d2fcdf0795a23d (127.30.2.195:41085)
I20260923 14:46:22.724041 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 14:46:22.726830 8075 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:46:22.725983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.30.2.254:44773)
0923 14:46:22.726098 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:22.726102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:22.726341 (+ 239us) server_negotiation.cc:407] Connection header received
0923 14:46:22.726424 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:22.726426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:22.726480 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:22.726551 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:22.726725 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60232: BlockingRecv error: recv got EOF from 127.0.0.1:60232 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:46:22.727041 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 8096
I20260923 14:46:22.733227 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 8227
I20260923 14:46:22.738787 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 8358
I20260923 14:46:22.744215 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 8005
2026-09-23T14:46:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (811 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:46:22 dist-test-slave-wjf2 krb5kdc[8496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:46:22 dist-test-slave-wjf2 krb5kdc[8496](info): set up 2 sockets
Sep 23 14:46:22 dist-test-slave-wjf2 krb5kdc[8496](info): commencing operation
krb5kdc: starting...
W20260923 14:46:23.782205 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:46:23 dist-test-slave-wjf2 krb5kdc[8496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:46:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:46:23.908280 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:33681
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:46:24.021760 8509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:24.022012 8509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:24.022058 8509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:24.026007 8509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:46:24.026062 8509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:24.026081 8509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:24.026098 8509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:46:24.026113 8509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:46:24.031203 8509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:24.032157 8509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:24.033099 8509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:24.041536 8514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:24.041569 8517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:24.041612 8509 server_base.cc:1034] running on GCE node
W20260923 14:46:24.041545 8515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:24.042287 8509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:24.043380 8509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:24.044540 8509 hybrid_clock.cc:648] HybridClock initialized: now 1790174784044526 us; error 29 us; skew 500 ppm
Sep 23 14:46:24 dist-test-slave-wjf2 krb5kdc[8496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:24.052424 8509 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:46:24.053397 8509 webserver.cc:460] Webserver started at http://127.30.2.254:34011/ using document root <none> and password file <none>
I20260923 14:46:24.054001 8509 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:24.054064 8509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:24.054306 8509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:24.056479 8509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "4dc7fae5ab5547d7ac19a82ebb0f94f7"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "3b54b0142aa19de7735a592cd7839a1a"
server_key_iv: "e94eceb3bd3135829c03eeb7a4bf9a61"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.057037 8509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "4dc7fae5ab5547d7ac19a82ebb0f94f7"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "3b54b0142aa19de7735a592cd7839a1a"
server_key_iv: "e94eceb3bd3135829c03eeb7a4bf9a61"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.060762 8509 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:46:24.063169 8524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:24.064404 8509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:46:24.064528 8509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "4dc7fae5ab5547d7ac19a82ebb0f94f7"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "3b54b0142aa19de7735a592cd7839a1a"
server_key_iv: "e94eceb3bd3135829c03eeb7a4bf9a61"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.064627 8509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:24.079928 8509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:24.083133 8509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:24.083341 8509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:24.090621 8509 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:33681
I20260923 14:46:24.090683 8576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:33681 every 8 connection(s)
I20260923 14:46:24.091553 8509 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:46:24.094380 8577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:24.094395 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 8509
I20260923 14:46:24.094528 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:46:24.094813 30731 external_mini_cluster.cc:1428] Setting key 117e9a3e008bb7cd59707306fda9b030
I20260923 14:46:24.101106 8577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7: Bootstrap starting.
I20260923 14:46:24.103885 8577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7: Neither blocks nor log segments found. Creating new log.
Sep 23 14:46:24 dist-test-slave-wjf2 krb5kdc[8496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:46:24.104628 8577 log.cc:824] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7: Log is configured to *not* fsync() on all Append() calls
I20260923 14:46:24.106771 8577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7: No bootstrap required, opened a new log
I20260923 14:46:24.109969 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.096247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.30.2.254:33681)
0923 14:46:24.096632 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:24.096647 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:24.096682 (+ 35us) server_negotiation.cc:407] Connection header received
0923 14:46:24.097528 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:24.097548 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:24.097870 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:24.098152 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:24.099052 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.100097 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.101231 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.101686 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.101802 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.104539 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:24.104545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:24.104557 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:24.104583 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:24.106732 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:24.107303 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:24.107309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:24.107315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:24.107453 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:24.107709 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:24.107712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:24.107713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:24.108228 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:24.108420 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:24.108653 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 14:46:24.108973 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":114,"threads_started":1}
I20260923 14:46:24.111313 8577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc7fae5ab5547d7ac19a82ebb0f94f7" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33681 } }
I20260923 14:46:24.111509 8577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:24.111567 8577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc7fae5ab5547d7ac19a82ebb0f94f7, State: Initialized, Role: FOLLOWER
I20260923 14:46:24.112058 8577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc7fae5ab5547d7ac19a82ebb0f94f7" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33681 } }
I20260923 14:46:24.112226 8577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:46:24.112285 8577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:46:24.112375 8577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:24.113222 8577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc7fae5ab5547d7ac19a82ebb0f94f7" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33681 } }
I20260923 14:46:24.113559 8577 leader_election.cc:302] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dc7fae5ab5547d7ac19a82ebb0f94f7; no voters:
I20260923 14:46:24.113863 8577 leader_election.cc:288] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:46:24.114002 8582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:46:24.114249 8582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc7fae5ab5547d7ac19a82ebb0f94f7, State: Running, Role: LEADER
I20260923 14:46:24.114569 8582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc7fae5ab5547d7ac19a82ebb0f94f7" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33681 } }
I20260923 14:46:24.115080 8577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:46:24.116281 8583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dc7fae5ab5547d7ac19a82ebb0f94f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc7fae5ab5547d7ac19a82ebb0f94f7" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33681 } } }
I20260923 14:46:24.116400 8583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [sys.catalog]: This master's current role is: LEADER
I20260923 14:46:24.116677 8584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dc7fae5ab5547d7ac19a82ebb0f94f7. Latest consensus state: current_term: 1 leader_uuid: "4dc7fae5ab5547d7ac19a82ebb0f94f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc7fae5ab5547d7ac19a82ebb0f94f7" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33681 } } }
I20260923 14:46:24.116827 8584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7 [sys.catalog]: This master's current role is: LEADER
I20260923 14:46:24.117502 8591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:46:24.120280 8591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:46:24.125696 8591 catalog_manager.cc:1347] Generated new cluster ID: 2d238f4f00534ed0a6363bb2a5f8c963
I20260923 14:46:24.125770 8591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:46:24.133302 8591 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:46:24.134195 8591 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:46:24.142719 8591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7: Generated new TSK 0
I20260923 14:46:24.143435 8591 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:46:24.182989 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33681
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:46:24.294785 8603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:24.295020 8603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:24.295061 8603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:24.299175 8603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:24.299221 8603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:24.299315 8603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:46:24.304414 8603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:24.305500 8603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:24.306540 8603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:24.316498 8608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:24.316514 8609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:24.316502 8611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:24.316555 8603 server_base.cc:1034] running on GCE node
I20260923 14:46:24.317224 8603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:24.317884 8603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:24.319047 8603 hybrid_clock.cc:648] HybridClock initialized: now 1790174784319001 us; error 55 us; skew 500 ppm
Sep 23 14:46:24 dist-test-slave-wjf2 krb5kdc[8496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:24.321812 8603 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:46:24.322737 8603 webserver.cc:460] Webserver started at http://127.30.2.193:36997/ using document root <none> and password file <none>
I20260923 14:46:24.323314 8603 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:24.323405 8603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:24.323637 8603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:24.325708 8603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "7bd1da9078764ad481f6d854bf59e383"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "525131da9096822c1bce17ef9bd602a4"
server_key_iv: "316acce47f37545b555f7ee4bbd01752"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.326264 8603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "7bd1da9078764ad481f6d854bf59e383"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "525131da9096822c1bce17ef9bd602a4"
server_key_iv: "316acce47f37545b555f7ee4bbd01752"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.330088 8603 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 14:46:24.332532 8618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:24.333750 8603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:46:24.333902 8603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "7bd1da9078764ad481f6d854bf59e383"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "525131da9096822c1bce17ef9bd602a4"
server_key_iv: "316acce47f37545b555f7ee4bbd01752"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.334013 8603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:24.368110 8603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:24.371414 8603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:24.371629 8603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:24.372300 8603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:24.373154 8603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:24.373226 8603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:24.373302 8603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:24.373353 8603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:24.382638 8603 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:34887
I20260923 14:46:24.382685 8731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:34887 every 8 connection(s)
I20260923 14:46:24.383584 8603 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:46:24.389353 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 8603
I20260923 14:46:24.389457 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:46:24.389750 30731 external_mini_cluster.cc:1428] Setting key 787b1bf0babca80631e43dc5b1fc288e
Sep 23 14:46:24 dist-test-slave-wjf2 krb5kdc[8496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:46:24.398463 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.385498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:42747 (local address 127.30.2.254:33681)
0923 14:46:24.385668 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:24.385673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:24.386515 (+ 842us) server_negotiation.cc:407] Connection header received
0923 14:46:24.387969 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:24.387972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:24.388020 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:24.388098 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:24.389321 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.389986 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.391371 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.391593 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.391672 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.394478 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:24.394485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:24.394491 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:24.394523 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:24.396281 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:24.396778 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:24.396785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:24.396789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:24.396888 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:24.397209 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:24.397215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:24.397219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:24.397500 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:24.397619 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:24.398111 (+ 492us) server_negotiation.cc:299] Negotiation successful
0923 14:46:24.398262 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 14:46:24.399339 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.385912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33681 (local address 127.30.2.193:42747)
0923 14:46:24.386365 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:24.386408 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:24.387717 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:24.388287 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:24.388294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:24.388634 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:24.389141 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.389162 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.390111 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.390115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.391252 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.391263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.391749 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.391756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.392019 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.392670 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:24.392696 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:24.394296 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:24.396391 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:24.396396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:24.396409 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:24.396657 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:24.396983 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:24.396986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:24.396987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:24.397114 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:24.397606 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:24.397611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:24.397856 (+ 245us) client_negotiation.cc:769] Sending connection context
0923 14:46:24.398016 (+ 160us) client_negotiation.cc:240] Negotiation successful
0923 14:46:24.398295 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:46:24.400650 8732 heartbeater.cc:347] Connected to a master server at 127.30.2.254:33681
I20260923 14:46:24.400904 8732 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:24.401435 8732 heartbeater.cc:511] Master 127.30.2.254:33681 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
I20260923 14:46:24.403113 8541 ts_manager.cc:194] Registered new tserver with Master: 7bd1da9078764ad481f6d854bf59e383 (127.30.2.193:34887)
I20260923 14:46:24.404317 8541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:42747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:46:24.423310 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33681
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:46:24.534801 8737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:24.535041 8737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:24.535081 8737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:24.539131 8737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:24.539183 8737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:24.539239 8737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:46:24.544420 8737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:24.545398 8737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:24.546386 8737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:46:24.555936 8743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:24.556041 8737 server_base.cc:1034] running on GCE node
W20260923 14:46:24.555999 8742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:24.556006 8745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:24.556766 8737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:24.557448 8737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:24.558656 8737 hybrid_clock.cc:648] HybridClock initialized: now 1790174784558614 us; error 55 us; skew 500 ppm
Sep 23 14:46:24 dist-test-slave-wjf2 krb5kdc[8496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:24.561460 8737 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:46:24.562299 8737 webserver.cc:460] Webserver started at http://127.30.2.194:44989/ using document root <none> and password file <none>
I20260923 14:46:24.562867 8737 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:24.562956 8737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:24.563213 8737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:24.565249 8737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "ee756208dbf23ecf87e7d94c3729b45f"
server_key_iv: "af993c4ec03baf7eab6230d33d298c36"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.565796 8737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "ee756208dbf23ecf87e7d94c3729b45f"
server_key_iv: "af993c4ec03baf7eab6230d33d298c36"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.569730 8737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:46:24.572083 8752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:24.573251 8737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:46:24.573379 8737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "ee756208dbf23ecf87e7d94c3729b45f"
server_key_iv: "af993c4ec03baf7eab6230d33d298c36"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.573484 8737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:24.589900 8737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:24.592609 8737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:24.592796 8737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:24.593345 8737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:24.594125 8737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:24.594193 8737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:24.594262 8737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:24.594304 8737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:24.602433 8737 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:42943
I20260923 14:46:24.602505 8865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:42943 every 8 connection(s)
I20260923 14:46:24.603349 8737 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:46:24.609553 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 8737
I20260923 14:46:24.609671 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:46:24.609936 30731 external_mini_cluster.cc:1428] Setting key c45f4822f1d814e5adcdf3661d039e75
Sep 23 14:46:24 dist-test-slave-wjf2 krb5kdc[8496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:46:24.617815 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.605090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:49751 (local address 127.30.2.254:33681)
0923 14:46:24.605237 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:24.605241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:24.606044 (+ 803us) server_negotiation.cc:407] Connection header received
0923 14:46:24.607457 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:24.607460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:24.607519 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:24.607626 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:24.608818 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.609351 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.610731 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.610958 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.611080 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.613666 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:24.613672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:24.613677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:24.613712 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:24.615412 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:24.616120 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:24.616124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:24.616126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:24.616213 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:24.616564 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:24.616567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:24.616569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:24.616845 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:24.616963 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:24.617512 (+ 549us) server_negotiation.cc:299] Negotiation successful
0923 14:46:24.617638 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:46:24.618669 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.605472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33681 (local address 127.30.2.194:49751)
0923 14:46:24.605898 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:24.605938 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:24.607246 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:24.607854 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:24.607864 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:24.608215 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:24.608672 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.608685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.609475 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.609479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.610606 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.610616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.611107 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.611112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.611280 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.611899 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:24.611915 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:24.613494 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:24.615543 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:24.615548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:24.615559 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:24.615959 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:24.616318 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:24.616321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:24.616324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:24.616459 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:24.616962 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:24.616969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:24.617263 (+ 294us) client_negotiation.cc:769] Sending connection context
0923 14:46:24.617449 (+ 186us) client_negotiation.cc:240] Negotiation successful
0923 14:46:24.617781 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:46:24.619894 8866 heartbeater.cc:347] Connected to a master server at 127.30.2.254:33681
I20260923 14:46:24.620113 8866 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:24.620591 8866 heartbeater.cc:511] Master 127.30.2.254:33681 requested a full tablet report, sending...
I20260923 14:46:24.621652 8541 ts_manager.cc:194] Registered new tserver with Master: 7ae2b5e331fa4aa299a7eaea0b7bba04 (127.30.2.194:42943)
I20260923 14:46:24.622299 8541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:49751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:46:24.639175 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33681
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:46:24.750628 8871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:46:24.750875 8871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:46:24.750916 8871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:46:24.755158 8871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:46:24.755214 8871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:46:24.755272 8871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:46:24.760459 8871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:46:24.761390 8871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:46:24.762354 8871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:46:24.772221 8871 server_base.cc:1034] running on GCE node
W20260923 14:46:24.772174 8876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:24.772145 8877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:46:24.772166 8879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:46:24.772760 8871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:46:24.773393 8871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:46:24.774546 8871 hybrid_clock.cc:648] HybridClock initialized: now 1790174784774532 us; error 26 us; skew 500 ppm
Sep 23 14:46:24 dist-test-slave-wjf2 krb5kdc[8496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:46:24.777387 8871 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:46:24.778295 8871 webserver.cc:460] Webserver started at http://127.30.2.195:34545/ using document root <none> and password file <none>
I20260923 14:46:24.778872 8871 fs_manager.cc:359] Metadata directory not provided
I20260923 14:46:24.778959 8871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:46:24.779178 8871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:46:24.781281 8871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "30c547377f1547e2b3d65cc607c7e86b"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "8cbf5db9075dbfaf71246648dfa63fd9"
server_key_iv: "94ab445df3f4b5adcc5c36746021e19d"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.781863 8871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "30c547377f1547e2b3d65cc607c7e86b"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "8cbf5db9075dbfaf71246648dfa63fd9"
server_key_iv: "94ab445df3f4b5adcc5c36746021e19d"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.785734 8871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260923 14:46:24.788311 8886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:24.789618 8871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:46:24.789754 8871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "30c547377f1547e2b3d65cc607c7e86b"
format_stamp: "Formatted at 2026-09-23 14:46:24 on dist-test-slave-wjf2"
server_key: "8cbf5db9075dbfaf71246648dfa63fd9"
server_key_iv: "94ab445df3f4b5adcc5c36746021e19d"
server_key_version: "encryptionkey@0"
I20260923 14:46:24.789867 8871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:46:24.801509 8871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:46:24.804883 8871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:46:24.805089 8871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:46:24.805701 8871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:46:24.806528 8871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:46:24.806598 8871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:24.806670 8871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:46:24.806721 8871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:46:24.817099 8871 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:41701
I20260923 14:46:24.817116 8999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:41701 every 8 connection(s)
I20260923 14:46:24.818073 8871 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:46:24.825304 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 8871
I20260923 14:46:24.825434 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:46:24.825726 30731 external_mini_cluster.cc:1428] Setting key a69577932d7795855b0e4c62f58c15f3
Sep 23 14:46:24 dist-test-slave-wjf2 krb5kdc[8496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:46:24.832157 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.819925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:57523 (local address 127.30.2.254:33681)
0923 14:46:24.820067 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:24.820071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:24.820990 (+ 919us) server_negotiation.cc:407] Connection header received
0923 14:46:24.822503 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:24.822506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:24.822575 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:24.822677 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:24.823888 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.824480 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.825992 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.826242 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.826343 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.828972 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:24.828975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:24.828977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:24.828999 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:24.830279 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:24.830758 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:24.830760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:24.830761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:24.830833 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:24.831128 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:24.831131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:24.831132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:24.831301 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:24.831374 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:24.831941 (+ 567us) server_negotiation.cc:299] Negotiation successful
0923 14:46:24.832037 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:46:24.833056 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.820358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33681 (local address 127.30.2.195:57523)
0923 14:46:24.820831 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:24.820875 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:24.822277 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:24.822849 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:24.822857 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:24.823212 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:46:24.823685 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.823698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.824603 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.824606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.825877 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.825887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.826414 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.826419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.826600 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.827252 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:46:24.827276 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:46:24.828817 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:46:24.830390 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:24.830394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:24.830405 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:24.830665 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:24.830907 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:46:24.830910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:46:24.830911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:46:24.831038 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:46:24.831417 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:46:24.831422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:46:24.831665 (+ 243us) client_negotiation.cc:769] Sending connection context
0923 14:46:24.831898 (+ 233us) client_negotiation.cc:240] Negotiation successful
0923 14:46:24.832243 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260923 14:46:24.834128 9000 heartbeater.cc:347] Connected to a master server at 127.30.2.254:33681
I20260923 14:46:24.834331 9000 heartbeater.cc:464] Registering TS with master...
I20260923 14:46:24.834791 9000 heartbeater.cc:511] Master 127.30.2.254:33681 requested a full tablet report, sending...
I20260923 14:46:24.835759 8541 ts_manager.cc:194] Registered new tserver with Master: 30c547377f1547e2b3d65cc607c7e86b (127.30.2.195:41701)
I20260923 14:46:24.836375 8541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:57523
I20260923 14:46:24.839669 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:46:24.848145 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.841551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.30.2.254:33681)
0923 14:46:24.841683 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:24.841698 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:24.841839 (+ 141us) server_negotiation.cc:407] Connection header received
0923 14:46:24.841927 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:24.841929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:24.841963 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:24.842025 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:46:24.842656 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.843136 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.843979 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.844177 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.844252 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.845494 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:46:24.845497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:46:24.845499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:46:24.845519 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:46:24.846709 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:24.847082 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:24.847084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:24.847085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:24.847151 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:46:24.847368 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:46:24.847370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:46:24.847371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:46:24.847552 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:46:24.847649 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:24.847883 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 14:46:24.847968 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:46:24.850721 8541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:46:24.853806 8541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:46:24.868511 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.862941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.30.2.195:41701)
0923 14:46:24.863757 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:24.863771 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:24.863796 (+ 25us) server_negotiation.cc:407] Connection header received
0923 14:46:24.863909 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:24.863914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:24.864114 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:24.864482 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:24.864636 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.865426 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.866885 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.867698 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.867825 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.867973 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:24.868241 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 14:46:24.868343 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":133,"threads_started":1}
I20260923 14:46:24.868530 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.862277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:41701 (local address 127.0.0.1:37682)
0923 14:46:24.862579 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:24.862615 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:24.862771 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:24.864283 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:24.864289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:24.864314 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:24.864555 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.864561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.865561 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.865567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.866751 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.866761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.867855 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.867861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.867988 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.868031 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 14:46:24.868156 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 14:46:24.868357 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":74,"thread_start_us":60,"threads_started":1}
I20260923 14:46:24.868538 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.862965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:34887 (local address 127.0.0.1:34454)
0923 14:46:24.863343 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:24.863357 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:24.863457 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:24.864216 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:24.864221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:24.864242 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:24.864539 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.864550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.866012 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.866017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.867045 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.867055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.867938 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.867941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.868196 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.868227 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 14:46:24.868294 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 14:46:24.868388 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":288,"thread_start_us":139,"threads_started":1}
I20260923 14:46:24.869179 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.863125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.30.2.193:34887)
0923 14:46:24.863737 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:24.863772 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:24.863810 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:46:24.863895 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:24.863899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:24.864053 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:24.864215 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:24.864683 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.865863 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.867162 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.867747 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.868012 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.868154 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:24.868895 (+ 741us) server_negotiation.cc:299] Negotiation successful
0923 14:46:24.869030 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":111,"threads_started":1}
I20260923 14:46:24.870343 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.862359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:42943 (local address 127.0.0.1:45140)
0923 14:46:24.863226 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:24.863242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:24.863329 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:24.864363 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:24.864365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:24.864381 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:24.864678 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.864684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.866136 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.866139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.867992 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.868005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.869272 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.869278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.870050 (+ 772us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.870074 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 14:46:24.870149 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 14:46:24.870217 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":698,"spinlock_wait_cycles":9472,"thread_start_us":94,"threads_started":1}
I20260923 14:46:24.870930 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.863437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.30.2.194:42943)
0923 14:46:24.863959 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:24.863969 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:24.863988 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:46:24.864050 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:24.864054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:24.864218 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:24.864495 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:24.864797 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.865991 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.868122 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.868971 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.869070 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.869201 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:24.870680 (+ 1479us) server_negotiation.cc:299] Negotiation successful
0923 14:46:24.870788 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":116,"threads_started":1}
I20260923 14:46:24.871814 8934 tablet_service.cc:1467] Processing CreateTablet for tablet fdec3a9dc5174aab837f58142edbde45 (DEFAULT_TABLE table=test-table [id=aae1d080fbb54e6fbb99ef47c2f8a4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:24.871927 8666 tablet_service.cc:1467] Processing CreateTablet for tablet fdec3a9dc5174aab837f58142edbde45 (DEFAULT_TABLE table=test-table [id=aae1d080fbb54e6fbb99ef47c2f8a4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:24.872879 8934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdec3a9dc5174aab837f58142edbde45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:24.872924 8666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdec3a9dc5174aab837f58142edbde45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:24.873559 8800 tablet_service.cc:1467] Processing CreateTablet for tablet fdec3a9dc5174aab837f58142edbde45 (DEFAULT_TABLE table=test-table [id=aae1d080fbb54e6fbb99ef47c2f8a4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:46:24.874464 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdec3a9dc5174aab837f58142edbde45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:46:24.879142 9017 tablet_bootstrap.cc:492] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b: Bootstrap starting.
I20260923 14:46:24.880491 9018 tablet_bootstrap.cc:492] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383: Bootstrap starting.
I20260923 14:46:24.881278 9019 tablet_bootstrap.cc:492] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04: Bootstrap starting.
I20260923 14:46:24.882045 9017 tablet_bootstrap.cc:654] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:24.882687 9018 tablet_bootstrap.cc:654] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:24.883409 9017 log.cc:824] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:46:24.883463 9019 tablet_bootstrap.cc:654] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04: Neither blocks nor log segments found. Creating new log.
I20260923 14:46:24.883912 9018 log.cc:824] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383: Log is configured to *not* fsync() on all Append() calls
I20260923 14:46:24.884341 9019 log.cc:824] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04: Log is configured to *not* fsync() on all Append() calls
I20260923 14:46:24.885493 9017 tablet_bootstrap.cc:492] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b: No bootstrap required, opened a new log
I20260923 14:46:24.885706 9017 ts_tablet_manager.cc:1397] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 14:46:24.885865 9018 tablet_bootstrap.cc:492] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383: No bootstrap required, opened a new log
I20260923 14:46:24.886047 9018 ts_tablet_manager.cc:1397] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:46:24.886658 9019 tablet_bootstrap.cc:492] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04: No bootstrap required, opened a new log
I20260923 14:46:24.886883 9019 ts_tablet_manager.cc:1397] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:46:24.889183 9017 raft_consensus.cc:348] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c547377f1547e2b3d65cc607c7e86b" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41701 } } peers { permanent_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42943 } } peers { permanent_uuid: "7bd1da9078764ad481f6d854bf59e383" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 34887 } }
I20260923 14:46:24.889468 9017 raft_consensus.cc:374] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:24.889545 9017 raft_consensus.cc:729] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c547377f1547e2b3d65cc607c7e86b, State: Initialized, Role: FOLLOWER
I20260923 14:46:24.890010 9017 consensus_queue.cc:260] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c547377f1547e2b3d65cc607c7e86b" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41701 } } peers { permanent_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42943 } } peers { permanent_uuid: "7bd1da9078764ad481f6d854bf59e383" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 34887 } }
I20260923 14:46:24.889901 9018 raft_consensus.cc:348] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c547377f1547e2b3d65cc607c7e86b" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41701 } } peers { permanent_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42943 } } peers { permanent_uuid: "7bd1da9078764ad481f6d854bf59e383" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 34887 } }
I20260923 14:46:24.890133 9018 raft_consensus.cc:374] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:24.890229 9018 raft_consensus.cc:729] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd1da9078764ad481f6d854bf59e383, State: Initialized, Role: FOLLOWER
I20260923 14:46:24.890667 9000 heartbeater.cc:503] Master 127.30.2.254:33681 was elected leader, sending a full tablet report...
I20260923 14:46:24.890667 9018 consensus_queue.cc:260] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c547377f1547e2b3d65cc607c7e86b" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41701 } } peers { permanent_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42943 } } peers { permanent_uuid: "7bd1da9078764ad481f6d854bf59e383" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 34887 } }
I20260923 14:46:24.890627 9019 raft_consensus.cc:348] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c547377f1547e2b3d65cc607c7e86b" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41701 } } peers { permanent_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42943 } } peers { permanent_uuid: "7bd1da9078764ad481f6d854bf59e383" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 34887 } }
I20260923 14:46:24.890843 9019 raft_consensus.cc:374] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:46:24.890895 9019 raft_consensus.cc:729] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ae2b5e331fa4aa299a7eaea0b7bba04, State: Initialized, Role: FOLLOWER
I20260923 14:46:24.891011 9017 ts_tablet_manager.cc:1428] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:46:24.891319 9019 consensus_queue.cc:260] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c547377f1547e2b3d65cc607c7e86b" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41701 } } peers { permanent_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42943 } } peers { permanent_uuid: "7bd1da9078764ad481f6d854bf59e383" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 34887 } }
I20260923 14:46:24.891582 9018 ts_tablet_manager.cc:1428] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:46:24.891928 8732 heartbeater.cc:503] Master 127.30.2.254:33681 was elected leader, sending a full tablet report...
I20260923 14:46:24.892210 9019 ts_tablet_manager.cc:1428] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260923 14:46:24.892531 8866 heartbeater.cc:503] Master 127.30.2.254:33681 was elected leader, sending a full tablet report...
I20260923 14:46:24.914690 9025 raft_consensus.cc:482] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:46:24.914937 9025 raft_consensus.cc:504] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c547377f1547e2b3d65cc607c7e86b" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41701 } } peers { permanent_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42943 } } peers { permanent_uuid: "7bd1da9078764ad481f6d854bf59e383" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 34887 } }
I20260923 14:46:24.916252 9025 leader_election.cc:288] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30c547377f1547e2b3d65cc607c7e86b (127.30.2.195:41701), 7bd1da9078764ad481f6d854bf59e383 (127.30.2.193:34887)
I20260923 14:46:24.920749 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.916581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:56407 (local address 127.30.2.193:34887)
0923 14:46:24.916701 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:24.916706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:24.916724 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:46:24.916841 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:24.916844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:24.916899 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:24.917002 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:24.917358 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.918057 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.919549 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.920159 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.920272 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.920328 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:24.920533 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 14:46:24.920602 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:46:24.920742 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.916496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:34887 (local address 127.30.2.194:56407)
0923 14:46:24.916611 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:24.916628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:24.916719 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:24.917036 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:24.917040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:24.917063 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:24.917218 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.917227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.918182 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.918185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.919420 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.919431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.920286 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.920289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.920372 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.920394 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:46:24.920492 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 14:46:24.920565 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":23}
I20260923 14:46:24.920873 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.916573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:56461 (local address 127.30.2.195:41701)
0923 14:46:24.916729 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:24.916733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:24.917164 (+ 431us) server_negotiation.cc:407] Connection header received
0923 14:46:24.917225 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:24.917227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:24.917266 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:24.917384 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:46:24.917746 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.918385 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.919729 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.920346 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.920445 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.920513 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:24.920693 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 14:46:24.920760 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 14:46:24.920907 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.916496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:41701 (local address 127.30.2.194:56461)
0923 14:46:24.917049 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:24.917061 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:46:24.917152 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:46:24.917521 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:46:24.917524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:46:24.917541 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:46:24.917635 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.917640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.918493 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.918497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.919623 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:46:24.919630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.920466 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.920472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:46:24.920572 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.920590 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 14:46:24.920663 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 14:46:24.920715 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":136,"thread_start_us":135,"threads_started":1}
I20260923 14:46:24.921527 8686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdec3a9dc5174aab837f58142edbde45" candidate_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd1da9078764ad481f6d854bf59e383" is_pre_election: true
I20260923 14:46:24.921639 8954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdec3a9dc5174aab837f58142edbde45" candidate_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c547377f1547e2b3d65cc607c7e86b" is_pre_election: true
I20260923 14:46:24.921856 8686 raft_consensus.cc:2454] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ae2b5e331fa4aa299a7eaea0b7bba04 in term 0.
I20260923 14:46:24.921880 8954 raft_consensus.cc:2454] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ae2b5e331fa4aa299a7eaea0b7bba04 in term 0.
I20260923 14:46:24.922341 8755 leader_election.cc:302] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30c547377f1547e2b3d65cc607c7e86b, 7ae2b5e331fa4aa299a7eaea0b7bba04, 7bd1da9078764ad481f6d854bf59e383; no voters:
I20260923 14:46:24.922662 9025 raft_consensus.cc:2781] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:46:24.922757 9025 raft_consensus.cc:482] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:46:24.922818 9025 raft_consensus.cc:3037] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:24.923873 9025 raft_consensus.cc:504] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c547377f1547e2b3d65cc607c7e86b" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41701 } } peers { permanent_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42943 } } peers { permanent_uuid: "7bd1da9078764ad481f6d854bf59e383" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 34887 } }
I20260923 14:46:24.924214 9025 leader_election.cc:288] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [CANDIDATE]: Term 1 election: Requested vote from peers 30c547377f1547e2b3d65cc607c7e86b (127.30.2.195:41701), 7bd1da9078764ad481f6d854bf59e383 (127.30.2.193:34887)
I20260923 14:46:24.924669 8686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdec3a9dc5174aab837f58142edbde45" candidate_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd1da9078764ad481f6d854bf59e383"
I20260923 14:46:24.924712 8954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdec3a9dc5174aab837f58142edbde45" candidate_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c547377f1547e2b3d65cc607c7e86b"
I20260923 14:46:24.924811 8954 raft_consensus.cc:3037] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:24.924804 8686 raft_consensus.cc:3037] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:46:24.926110 8686 raft_consensus.cc:2454] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ae2b5e331fa4aa299a7eaea0b7bba04 in term 1.
I20260923 14:46:24.926146 8954 raft_consensus.cc:2454] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ae2b5e331fa4aa299a7eaea0b7bba04 in term 1.
I20260923 14:46:24.926419 8754 leader_election.cc:302] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ae2b5e331fa4aa299a7eaea0b7bba04, 7bd1da9078764ad481f6d854bf59e383; no voters:
I20260923 14:46:24.926659 9025 raft_consensus.cc:2781] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:46:24.926903 9025 raft_consensus.cc:686] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 1 LEADER]: Becoming Leader. State: Replica: 7ae2b5e331fa4aa299a7eaea0b7bba04, State: Running, Role: LEADER
I20260923 14:46:24.927295 9025 consensus_queue.cc:237] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c547377f1547e2b3d65cc607c7e86b" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41701 } } peers { permanent_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42943 } } peers { permanent_uuid: "7bd1da9078764ad481f6d854bf59e383" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 34887 } }
I20260923 14:46:24.930521 8539 catalog_manager.cc:5495] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ae2b5e331fa4aa299a7eaea0b7bba04 (127.30.2.194). New cstate: current_term: 1 leader_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c547377f1547e2b3d65cc607c7e86b" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ae2b5e331fa4aa299a7eaea0b7bba04" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bd1da9078764ad481f6d854bf59e383" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 34887 } health_report { overall_health: UNKNOWN } } }
I20260923 14:46:24.951598 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.947992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.30.2.194:42943)
0923 14:46:24.948155 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 14:46:24.948181 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 14:46:24.948199 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:46:24.948364 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:46:24.948367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:46:24.948421 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:46:24.948511 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:46:24.948733 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.949234 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.950112 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:46:24.950338 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:46:24.950418 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:46:24.950725 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:46:24.951128 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:46:24.951243 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 14:46:24.951359 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 14:46:24.951407 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:46:24.957770 8686 raft_consensus.cc:1261] T fdec3a9dc5174aab837f58142edbde45 P 7bd1da9078764ad481f6d854bf59e383 [term 1 FOLLOWER]: Refusing update from remote peer 7ae2b5e331fa4aa299a7eaea0b7bba04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:46:24.957780 8954 raft_consensus.cc:1261] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b [term 1 FOLLOWER]: Refusing update from remote peer 7ae2b5e331fa4aa299a7eaea0b7bba04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:46:24.958534 9027 consensus_queue.cc:1035] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c547377f1547e2b3d65cc607c7e86b" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:24.958729 9025 consensus_queue.cc:1035] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd1da9078764ad481f6d854bf59e383" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:46:24.966569 9031 mvcc.cc:205] Tried to move back new op lower bound from 7332555919178821632 to 7332555919065391104. Current Snapshot: MvccSnapshot[applied={T|T < 7332555919178821632}]
I20260923 14:46:24.966733 9032 mvcc.cc:205] Tried to move back new op lower bound from 7332555919178821632 to 7332555919065391104. Current Snapshot: MvccSnapshot[applied={T|T < 7332555919178821632}]
I20260923 14:46:24.976312 8540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:46:24.976491 8540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:46:24.979132 8540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dc7fae5ab5547d7ac19a82ebb0f94f7: Sending DeleteTablet for 3 replicas of tablet fdec3a9dc5174aab837f58142edbde45
I20260923 14:46:24.979918 8934 tablet_service.cc:1514] Processing DeleteTablet for tablet fdec3a9dc5174aab837f58142edbde45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:24 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:37682
I20260923 14:46:24.979969 8666 tablet_service.cc:1514] Processing DeleteTablet for tablet fdec3a9dc5174aab837f58142edbde45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:24 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:34454
I20260923 14:46:24.980499 9040 tablet_replica.cc:330] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b: stopping tablet replica
I20260923 14:46:24.980549 8800 tablet_service.cc:1514] Processing DeleteTablet for tablet fdec3a9dc5174aab837f58142edbde45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:46:24 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:45140
I20260923 14:46:24.980655 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 8603
I20260923 14:46:24.981703 9040 raft_consensus.cc:2229] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:46:24.982089 9040 raft_consensus.cc:2258] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:24.983991 9042 tablet_replica.cc:330] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04: stopping tablet replica
I20260923 14:46:24.984045 9040 ts_tablet_manager.cc:1905] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:24.984334 9042 raft_consensus.cc:2229] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:46:24.984771 9042 raft_consensus.cc:2258] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:46:24.986280 9042 ts_tablet_manager.cc:1905] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:46:24.986927 9040 ts_tablet_manager.cc:1918] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:46:24.987074 9040 log.cc:1196] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/fdec3a9dc5174aab837f58142edbde45
I20260923 14:46:24.987468 9040 ts_tablet_manager.cc:1939] T fdec3a9dc5174aab837f58142edbde45 P 30c547377f1547e2b3d65cc607c7e86b: Deleting consensus metadata
I20260923 14:46:24.988893 8527 catalog_manager.cc:4841] TS 30c547377f1547e2b3d65cc607c7e86b (127.30.2.195:41701): tablet fdec3a9dc5174aab837f58142edbde45 (table test-table [id=aae1d080fbb54e6fbb99ef47c2f8a4bc]) successfully deleted
I20260923 14:46:24.989130 9042 ts_tablet_manager.cc:1918] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:46:24.989300 9042 log.cc:1196] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/fdec3a9dc5174aab837f58142edbde45
W20260923 14:46:24.989578 8526 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:34887 (error 108)
I20260923 14:46:24.989691 9042 ts_tablet_manager.cc:1939] T fdec3a9dc5174aab837f58142edbde45 P 7ae2b5e331fa4aa299a7eaea0b7bba04: Deleting consensus metadata
I20260923 14:46:24.989763 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 8737
I20260923 14:46:24.991828 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.991468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:34887 (local address 127.0.0.1:34460)
0923 14:46:24.991615 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:24.991696 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:34887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 14:46:24.992004 8526 catalog_manager.cc:4584] TS 7bd1da9078764ad481f6d854bf59e383 (127.30.2.193:34887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fdec3a9dc5174aab837f58142edbde45: Network error: Client connection negotiation failed: client connection to 127.30.2.193:34887: connect: Connection refused (error 111)
I20260923 14:46:24.996403 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 8871
I20260923 14:46:24.997248 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:46:24.996617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:42943 (local address 127.0.0.1:45158)
0923 14:46:24.997087 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 14:46:24.997146 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.194:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369}
I20260923 14:46:25.002368 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 8509
2026-09-23T14:46:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2259 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:46:25 dist-test-slave-wjf2 krb5kdc[9046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:46:25 dist-test-slave-wjf2 krb5kdc[9046](info): set up 2 sockets
Sep 23 14:46:25 dist-test-slave-wjf2 krb5kdc[9046](info): commencing operation
krb5kdc: starting...
W20260923 14:46:26.042362 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:46:26 dist-test-slave-wjf2 krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:46:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/rangeradmin_127.30.2.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/rangeradmin_127.30.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/rangerlookup_127.30.2.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/rangerlookup_127.30.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/HTTP_127.30.2.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/HTTP_127.30.2.212@KRBTEST.COM.keytab.
I20260923 14:46:26.214377 30731 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknhhgz8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/postgres -l logfile start
2026-09-23 14:46:28.889 UTC [9076] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 14:46:28.889 UTC [9076] LOG: listening on IPv4 address "127.30.2.212", port 37629
2026-09-23 14:46:28.893 UTC [9076] LOG: listening on Unix socket "/tmp/.s.PGSQL.37629"
2026-09-23 14:46:28.914 UTC [9079] LOG: database system was shut down at 2026-09-23 14:46:27 UTC
2026-09-23 14:46:28.918 UTC [9076] LOG: database system is ready to accept connections
I20260923 14:46:29.893949 30731 mini_postgres.cc:96] Postgres bound to 37629
2026-09-23 14:46:29.902 UTC [9087] FATAL: database "slave" does not exist
127.30.2.212:37629 - accepting connections
I20260923 14:46:29.902727 30731 mini_ranger.cc:161] Starting Ranger...
I20260923 14:46:29.918326 30731 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 14:46:30.439414 30731 mini_ranger.cc:87] Created ranger Postgres database
I20260923 14:46:30.439519 30731 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-admin
2026-09-23 14:46:30,964 [I] DB FLAVOR :POSTGRES
2026-09-23 14:46:30,965 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 14:46:30,965 [I] Checking connection..
2026-09-23 14:46:30,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 14:46:31,270 [I] Checking connection passed.
2026-09-23 14:46:31,271 [I] --------- Verifying version history table ---------
2026-09-23 14:46:31,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:46:31,516 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 14:46:31,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:46:31,754 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 14:46:31,755 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 14:46:31,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 14:46:32.011 UTC [9151] WARNING: there is no transaction in progress
2026-09-23 14:46:32,018 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 14:46:32,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:46:32,289 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 14:46:32,289 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 14:46:32,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 14:46:32,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 14:46:32,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wjf2.c.gcp-upstream.internal','N') ;"
2026-09-23 14:46:32,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 14:46:33,226 [I] Table x_portal_user does not exist in database ranger
2026-09-23 14:46:33,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 14:46:33,489 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 14:46:33,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 14:46:33,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 14:46:33,962 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 14:46:33,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 14:46:34.925 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:34.942 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:34.958 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:34.974 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:34.988 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35.008 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35.062 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35.069 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35.084 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35.094 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35.104 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35.117 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35.127 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35.134 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35.140 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35.147 UTC [9259] WARNING: there is no transaction in progress
2026-09-23 14:46:35,822 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 14:46:35,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 14:46:36,064 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 14:46:36,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 14:46:36,290 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 14:46:36,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 14:46:36,517 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 14:46:36,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 14:46:36,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 14:46:37,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wjf2.c.gcp-upstream.internal';"
2026-09-23 14:46:37,283 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 14:46:37,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 14:46:37,518 [I] DB_PATCHES have already been applied
I20260923 14:46:37.524698 30731 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknhhgz8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 14:46:37.524787 30731 mini_ranger.cc:193] Using host: 127.30.2.212
I20260923 14:46:37.528738 30731 mini_ranger.cc:227] Ranger admin URL: http://127.30.2.212:50525
Sep 23, 2026 2:46:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 2:46:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 2:46:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 2:46:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 2:46:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 2:46:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 2:46:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 14:46:39.766053 30731 mini_ranger.cc:160] Time spent starting Ranger: real 9.863s user 0.004s sys 0.008s
Sep 23, 2026 2:46:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 2:46:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 2:46:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 2:46:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 2:47:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 14:47:01.223649 30731 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/rangerkms_127.30.2.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/rangerkms_127.30.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.2.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.2.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.2.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/HTTP_127.30.2.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.2.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/HTTP_127.30.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 14:47:01 dist-test-slave-wjf2 krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174821, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 14:47:01.319329 30731 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 14:47:01.329485 30731 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 14:47:02.267771 30731 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 14:47:02.267902 30731 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-kms
I20260923 14:47:02.268007 30731 mini_ranger_kms.cc:227] Using postgres at 127.30.2.212:37629
2026-09-23 14:47:02,744 [I] DB FLAVOR :POSTGRES
2026-09-23 14:47:02,745 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 14:47:02,745 [I] Checking connection
2026-09-23 14:47:02,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 14:47:02,994 [I] connection success
2026-09-23 14:47:02,994 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 14:47:02,994 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 14:47:02,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 14:47:03,255 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 14:47:03,255 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 14:47:03,256 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 14:47:03,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:37629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 14:47:03,501 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 14:47:04.029160 30731 mini_ranger_kms.cc:310] Created kms service
I20260923 14:47:04.233287 30731 mini_ranger_kms.cc:326] Created kudu user
I20260923 14:47:04.276566 30731 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 14:47:04 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174821, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:04.733880 30731 mini_ranger_kms.cc:384] Added ranger policy
I20260923 14:47:04.734023 30731 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 14:47:04.734045 30731 mini_ranger_kms.cc:240] Using host: 127.30.2.212
I20260923 14:47:04.737774 30731 mini_ranger_kms.cc:276] Ranger KMS PID: 9441
I20260923 14:47:04.737920 30731 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.2.212:51599
Sep 23, 2026 2:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 2:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 2:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 2:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 2:47:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 2:47:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 14:47:07.009754 30731 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.690s user 0.005s sys 0.224s
I20260923 14:47:07.009918 30731 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 14:47:07.010046 30731 mini_ranger_kms.cc:401] 127.30.2.212:51599/kms/v1/keys
Sep 23 14:47:07 dist-test-slave-wjf2 krb5kdc[9046](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790174827, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.2.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 14:47:07 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174827, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.2.212@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
Sep 23 14:47:08 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174827, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.2.212@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
Sep 23 14:47:10 dist-test-slave-wjf2 krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:47:10.286890 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:43985
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:43985
--ranger_config_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:51599/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:47:10.401628 9491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:47:10.401904 9491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:47:10.401950 9491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:47:10.406090 9491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:47:10.406154 9491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:47:10.406177 9491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:47:10.406196 9491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:47:10.406247 9491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:47:10.411810 9491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.2.212:51599/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:43985
--ranger_config_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:43985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:47:10.413013 9491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:47:10.414129 9491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:47:10.423617 9499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:10.423697 9496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:10.424003 9491 server_base.cc:1034] running on GCE node
W20260923 14:47:10.423702 9497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:10.424935 9491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:47:10.426347 9491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:47:10.427567 9491 hybrid_clock.cc:648] HybridClock initialized: now 1790174830427529 us; error 43 us; skew 500 ppm
Sep 23 14:47:10 dist-test-slave-wjf2 krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174830, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:47:10.430776 9491 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:47:10.431813 9491 webserver.cc:460] Webserver started at http://127.30.2.254:37727/ using document root <none> and password file <none>
I20260923 14:47:10.432359 9491 fs_manager.cc:359] Metadata directory not provided
I20260923 14:47:10.432442 9491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:47:10.432652 9491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:47:10 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174830, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:10.480351 9491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "5d448b253efd43e68b6cbe18913af535"
format_stamp: "Formatted at 2026-09-23 14:47:10 on dist-test-slave-wjf2"
server_key: "1663755bf52f3e17bad2ae8f504a7f11"
server_key_iv: "9ddbb89bb22a67989f31163920b308f9"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:10.480976 9491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "5d448b253efd43e68b6cbe18913af535"
format_stamp: "Formatted at 2026-09-23 14:47:10 on dist-test-slave-wjf2"
server_key: "1663755bf52f3e17bad2ae8f504a7f11"
server_key_iv: "9ddbb89bb22a67989f31163920b308f9"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:10.485060 9491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 14:47:10.499403 9506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:10.500903 9491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:47:10.501057 9491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "5d448b253efd43e68b6cbe18913af535"
format_stamp: "Formatted at 2026-09-23 14:47:10 on dist-test-slave-wjf2"
server_key: "1663755bf52f3e17bad2ae8f504a7f11"
server_key_iv: "9ddbb89bb22a67989f31163920b308f9"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:10.501209 9491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:47:10.515187 9491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:47:10.518318 9491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:47:10.518540 9491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:47:10.526887 9491 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:43985
I20260923 14:47:10.526901 9558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:43985 every 8 connection(s)
I20260923 14:47:10.527997 9491 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:47:10.531504 9559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:47:10.534505 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 9491
I20260923 14:47:10.534665 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:47:10.534967 30731 external_mini_cluster.cc:1428] Setting key 3c495f71df05143d90f884a57a60553b
I20260923 14:47:10.538901 9559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535: Bootstrap starting.
I20260923 14:47:10.541766 9559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535: Neither blocks nor log segments found. Creating new log.
I20260923 14:47:10.542661 9559 log.cc:824] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535: Log is configured to *not* fsync() on all Append() calls
Sep 23 14:47:10 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:47:10.545094 9559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535: No bootstrap required, opened a new log
I20260923 14:47:10.548998 9559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d448b253efd43e68b6cbe18913af535" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43985 } }
I20260923 14:47:10.549261 9559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:47:10.549345 9559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d448b253efd43e68b6cbe18913af535, State: Initialized, Role: FOLLOWER
I20260923 14:47:10.549841 9559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d448b253efd43e68b6cbe18913af535" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43985 } }
I20260923 14:47:10.549988 9559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:47:10.550071 9559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:47:10.550076 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:10.536555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.30.2.254:43985)
0923 14:47:10.536999 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:10.537021 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:10.537065 (+ 44us) server_negotiation.cc:407] Connection header received
0923 14:47:10.537739 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:10.537757 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:10.538101 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:10.538452 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:47:10.539296 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:10.540384 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:10.541298 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:10.541775 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:10.541965 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:10.544834 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:47:10.544840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:47:10.544854 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:47:10.544880 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:47:10.547105 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:10.547585 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:10.547591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:10.547601 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:10.547703 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:10.548033 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:10.548036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:10.548038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:10.548414 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:47:10.548643 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:10.548889 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 14:47:10.549173 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":145,"threads_started":1}
I20260923 14:47:10.550175 9559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:47:10.551208 9559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d448b253efd43e68b6cbe18913af535" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43985 } }
I20260923 14:47:10.551563 9559 leader_election.cc:302] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d448b253efd43e68b6cbe18913af535; no voters:
I20260923 14:47:10.551899 9559 leader_election.cc:288] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:47:10.551982 9564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:47:10.552755 9564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [term 1 LEADER]: Becoming Leader. State: Replica: 5d448b253efd43e68b6cbe18913af535, State: Running, Role: LEADER
I20260923 14:47:10.553146 9564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d448b253efd43e68b6cbe18913af535" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43985 } }
I20260923 14:47:10.553269 9559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:47:10.554231 9564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d448b253efd43e68b6cbe18913af535. Latest consensus state: current_term: 1 leader_uuid: "5d448b253efd43e68b6cbe18913af535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d448b253efd43e68b6cbe18913af535" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43985 } } }
I20260923 14:47:10.554428 9564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [sys.catalog]: This master's current role is: LEADER
I20260923 14:47:10.554749 9566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d448b253efd43e68b6cbe18913af535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d448b253efd43e68b6cbe18913af535" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43985 } } }
I20260923 14:47:10.554838 9566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535 [sys.catalog]: This master's current role is: LEADER
I20260923 14:47:10.554940 9573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:47:10.558156 9573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:47:10.558653 9559 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 14:47:10.564108 9573 catalog_manager.cc:1347] Generated new cluster ID: d7ddfc0b152241eca52fb2cfae5457d9
I20260923 14:47:10.564251 9573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:47:10.573251 9573 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:47:10.575158 9573 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:47:10.582626 9573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535: Generated new TSK 0
I20260923 14:47:10.583500 9573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:47:11.320621 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:51599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43985
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:47:11.494277 9604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:47:11.494697 9604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:47:11.494796 9604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:47:11.500828 9604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:47:11.500913 9604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:47:11.501019 9604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:47:11.509064 9604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.2.212:51599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:47:11.510618 9604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:47:11.512075 9604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:47:11.524755 9614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:11.525192 9612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:11.525614 9611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:11.526319 9604 server_base.cc:1034] running on GCE node
I20260923 14:47:11.526808 9604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:47:11.527653 9604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:47:11.528851 9604 hybrid_clock.cc:648] HybridClock initialized: now 1790174831528790 us; error 77 us; skew 500 ppm
Sep 23 14:47:11 dist-test-slave-wjf2 krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174831, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:47:11.531646 9604 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:47:11.532482 9604 webserver.cc:460] Webserver started at http://127.30.2.193:40761/ using document root <none> and password file <none>
I20260923 14:47:11.533067 9604 fs_manager.cc:359] Metadata directory not provided
I20260923 14:47:11.533164 9604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:47:11.533373 9604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:47:11 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174831, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:11.557441 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "0ea97e0c85354ead8db9cfe554154004"
format_stamp: "Formatted at 2026-09-23 14:47:11 on dist-test-slave-wjf2"
server_key: "15c2dcc5bc621c5f9d060ad5ee5a2392"
server_key_iv: "1dda6f69a3aaf0cf5c40557f4f8014f9"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:11.558177 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "0ea97e0c85354ead8db9cfe554154004"
format_stamp: "Formatted at 2026-09-23 14:47:11 on dist-test-slave-wjf2"
server_key: "15c2dcc5bc621c5f9d060ad5ee5a2392"
server_key_iv: "1dda6f69a3aaf0cf5c40557f4f8014f9"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:11.563169 9604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 14:47:11.580615 9621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:11.582302 9604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:47:11.582430 9604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "0ea97e0c85354ead8db9cfe554154004"
format_stamp: "Formatted at 2026-09-23 14:47:11 on dist-test-slave-wjf2"
server_key: "15c2dcc5bc621c5f9d060ad5ee5a2392"
server_key_iv: "1dda6f69a3aaf0cf5c40557f4f8014f9"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:11.582574 9604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:47:11.605623 9604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:47:11.609746 9604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:47:11.609956 9604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:47:11.610715 9604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:47:11.611896 9604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:47:11.611966 9604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:11.612033 9604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:47:11.612071 9604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:11.623513 9604 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:35589
I20260923 14:47:11.624821 9604 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:47:11.630142 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 9604
I20260923 14:47:11.630275 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:47:11.630648 30731 external_mini_cluster.cc:1428] Setting key 3fe8f6ef96483675b72c20ffc47009b8
I20260923 14:47:11.629840 9734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:35589 every 8 connection(s)
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
Sep 23 14:47:11 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174831, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:47:11.673883 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:51599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43985
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:47:11.678215 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:11.637199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:43677 (local address 127.30.2.254:43985)
0923 14:47:11.639881 (+ 2682us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:11.639885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:11.639902 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:47:11.639947 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:11.639952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:11.640003 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:11.641256 (+ 1253us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:47:11.663926 (+ 22670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:11.664911 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:11.667328 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:11.667746 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:11.667885 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:11.671296 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:47:11.671301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:47:11.671303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:47:11.671338 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:47:11.673339 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:11.676249 (+ 2910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:11.676253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:11.676255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:11.676354 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:11.676803 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:11.676807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:11.676809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:11.677052 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:47:11.677144 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:11.677865 (+ 721us) server_negotiation.cc:299] Negotiation successful
0923 14:47:11.678004 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2188,"thread_start_us":92,"threads_started":1}
I20260923 14:47:11.679744 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:11.635949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:43985 (local address 127.30.2.193:43677)
0923 14:47:11.637484 (+ 1535us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:11.637547 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:11.638975 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:11.640341 (+ 1366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:11.640351 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:11.640722 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:47:11.641449 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:11.641472 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:11.665135 (+ 23663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:11.665141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:11.667043 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:11.667060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:11.667939 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:11.667945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:11.668267 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:11.669110 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:47:11.669134 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:47:11.671103 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:47:11.675635 (+ 4532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:11.675644 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:11.675664 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:11.676114 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:11.676475 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:11.676479 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:11.676481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:11.676680 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:11.677229 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:47:11.677236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:47:11.677568 (+ 332us) client_negotiation.cc:769] Sending connection context
0923 14:47:11.678620 (+ 1052us) client_negotiation.cc:240] Negotiation successful
0923 14:47:11.678930 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"thread_start_us":227,"threads_started":1}
I20260923 14:47:11.681488 9735 heartbeater.cc:347] Connected to a master server at 127.30.2.254:43985
I20260923 14:47:11.681743 9735 heartbeater.cc:464] Registering TS with master...
I20260923 14:47:11.682334 9735 heartbeater.cc:511] Master 127.30.2.254:43985 requested a full tablet report, sending...
I20260923 14:47:11.684263 9523 ts_manager.cc:194] Registered new tserver with Master: 0ea97e0c85354ead8db9cfe554154004 (127.30.2.193:35589)
I20260923 14:47:11.685883 9523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:43677
Sep 23 14:47:11 dist-test-slave-wjf2 krb5kdc[9046](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790174831, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 14:47:11.843894 9743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:47:11.844166 9743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:47:11.844238 9743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:47:11.849102 9743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:47:11.849198 9743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:47:11.849303 9743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:47:11.855103 9743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.2.212:51599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:47:11.856460 9743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:47:11.857762 9743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:47:11.870610 9750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:11.871097 9752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:11.871366 9749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:11.872654 9743 server_base.cc:1034] running on GCE node
I20260923 14:47:11.873180 9743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:47:11.874035 9743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:47:11.875243 9743 hybrid_clock.cc:648] HybridClock initialized: now 1790174831875175 us; error 82 us; skew 500 ppm
Sep 23 14:47:11 dist-test-slave-wjf2 krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174831, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:47:11.878958 9743 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:47:11.880259 9743 webserver.cc:460] Webserver started at http://127.30.2.194:35021/ using document root <none> and password file <none>
I20260923 14:47:11.881054 9743 fs_manager.cc:359] Metadata directory not provided
I20260923 14:47:11.881147 9743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:47:11.881387 9743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:47:11 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174831, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:11.904099 9743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "fd9228f437d2462eace71a6f2bbebda7"
format_stamp: "Formatted at 2026-09-23 14:47:11 on dist-test-slave-wjf2"
server_key: "5878b1c82a558def6ec9318c0f7e6708"
server_key_iv: "bafa0bd4c5ef58fcad0b39eb9081fc9a"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:11.905025 9743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "fd9228f437d2462eace71a6f2bbebda7"
format_stamp: "Formatted at 2026-09-23 14:47:11 on dist-test-slave-wjf2"
server_key: "5878b1c82a558def6ec9318c0f7e6708"
server_key_iv: "bafa0bd4c5ef58fcad0b39eb9081fc9a"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:11.910457 9743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 14:47:11.928082 9759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:11.929869 9743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:47:11.930027 9743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "fd9228f437d2462eace71a6f2bbebda7"
format_stamp: "Formatted at 2026-09-23 14:47:11 on dist-test-slave-wjf2"
server_key: "5878b1c82a558def6ec9318c0f7e6708"
server_key_iv: "bafa0bd4c5ef58fcad0b39eb9081fc9a"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:11.930197 9743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:47:11.954558 9743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:47:11.958496 9743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:47:11.958694 9743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:47:11.959375 9743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:47:11.960466 9743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:47:11.960536 9743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:11.960601 9743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:47:11.960636 9743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:11.974905 9743 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:34055
I20260923 14:47:11.977206 9872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:34055 every 8 connection(s)
I20260923 14:47:11.977694 9743 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:47:11.978180 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 9743
I20260923 14:47:11.978308 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:47:11.978636 30731 external_mini_cluster.cc:1428] Setting key 72529be2007fa7c544e31ba625544d22
Sep 23 14:47:11 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174831, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:47:11.998670 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:11.982806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:60935 (local address 127.30.2.254:43985)
0923 14:47:11.982930 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:11.982936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:11.983447 (+ 511us) server_negotiation.cc:407] Connection header received
0923 14:47:11.986124 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:11.986128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:11.986194 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:11.986250 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:47:11.988071 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:11.988870 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:11.990659 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:11.991027 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:11.991111 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:11.994224 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:47:11.994228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:47:11.994232 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:47:11.994261 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:47:11.996065 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:11.996765 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:11.996769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:11.996771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:11.996860 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:11.997318 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:11.997322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:11.997324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:11.997553 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:47:11.997644 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:11.998365 (+ 721us) server_negotiation.cc:299] Negotiation successful
0923 14:47:11.998501 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:47:12.000379 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:11.981739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:43985 (local address 127.30.2.194:60935)
0923 14:47:11.983297 (+ 1558us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:11.983354 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:11.985870 (+ 2516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:11.986655 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:11.986664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:11.987088 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:47:11.987825 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:11.987848 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:11.989034 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:11.989037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:11.990515 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:11.990527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:11.991343 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:11.991346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:11.991532 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:11.992273 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:47:11.992296 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:47:11.994003 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:47:11.996261 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:11.996267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:11.996282 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:11.996638 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:11.996979 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:11.996983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:11.996985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:11.997195 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:11.997716 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:47:11.997723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:47:11.998045 (+ 322us) client_negotiation.cc:769] Sending connection context
0923 14:47:11.998913 (+ 868us) client_negotiation.cc:240] Negotiation successful
0923 14:47:11.999271 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"thread_start_us":137,"threads_started":1}
I20260923 14:47:12.001762 9873 heartbeater.cc:347] Connected to a master server at 127.30.2.254:43985
I20260923 14:47:12.002034 9873 heartbeater.cc:464] Registering TS with master...
I20260923 14:47:12.002643 9873 heartbeater.cc:511] Master 127.30.2.254:43985 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
I20260923 14:47:12.003845 9523 ts_manager.cc:194] Registered new tserver with Master: fd9228f437d2462eace71a6f2bbebda7 (127.30.2.194:34055)
I20260923 14:47:12.004701 9523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:60935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:47:12.026855 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:51599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43985
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:47:12.212302 9878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:47:12.212709 9878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:47:12.212801 9878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:47:12.218636 9878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:47:12.218727 9878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:47:12.218833 9878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:47:12.226853 9878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.2.212:51599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:47:12.228358 9878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:47:12.229796 9878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:47:12.243256 9884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:12.243387 9883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:12.243870 9886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:12.245014 9878 server_base.cc:1034] running on GCE node
I20260923 14:47:12.245644 9878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:47:12.246434 9878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:47:12.247618 9878 hybrid_clock.cc:648] HybridClock initialized: now 1790174832247552 us; error 84 us; skew 500 ppm
Sep 23 14:47:12 dist-test-slave-wjf2 krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174832, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:47:12.251449 9878 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:47:12.252580 9878 webserver.cc:460] Webserver started at http://127.30.2.195:37977/ using document root <none> and password file <none>
I20260923 14:47:12.253340 9878 fs_manager.cc:359] Metadata directory not provided
I20260923 14:47:12.253433 9878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:47:12.253695 9878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:47:12 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174832, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:12.281929 9878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "bf787c48a3bf4483a70f8a1f9e73713c"
format_stamp: "Formatted at 2026-09-23 14:47:12 on dist-test-slave-wjf2"
server_key: "56fa5add47effc6fb1025be732c40ed4"
server_key_iv: "81ed5f52323d2da772bb019e354f65a4"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:12.282747 9878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "bf787c48a3bf4483a70f8a1f9e73713c"
format_stamp: "Formatted at 2026-09-23 14:47:12 on dist-test-slave-wjf2"
server_key: "56fa5add47effc6fb1025be732c40ed4"
server_key_iv: "81ed5f52323d2da772bb019e354f65a4"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:12.290406 9878 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 14:47:12.314348 9893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:12.316067 9878 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 14:47:12.316201 9878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "bf787c48a3bf4483a70f8a1f9e73713c"
format_stamp: "Formatted at 2026-09-23 14:47:12 on dist-test-slave-wjf2"
server_key: "56fa5add47effc6fb1025be732c40ed4"
server_key_iv: "81ed5f52323d2da772bb019e354f65a4"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:12.316339 9878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:47:12.347831 9878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:47:12.352191 9878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:47:12.352499 9878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:47:12.353438 9878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:47:12.354770 9878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:47:12.354887 9878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:12.354996 9878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:47:12.355072 9878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:12.374356 9878 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:37539
I20260923 14:47:12.376884 10006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:37539 every 8 connection(s)
I20260923 14:47:12.375744 9878 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:47:12.379587 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 9878
I20260923 14:47:12.379707 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:47:12.380071 30731 external_mini_cluster.cc:1428] Setting key 7cd070f76dc5d6459b2871cd18ee24fe
Sep 23 14:47:12 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174832, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:47:12.397647 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.383505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:38069 (local address 127.30.2.254:43985)
0923 14:47:12.383641 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:12.383647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:12.384139 (+ 492us) server_negotiation.cc:407] Connection header received
0923 14:47:12.385726 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:12.385730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:12.385782 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:12.385834 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:47:12.387514 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.388330 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.390028 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.390389 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.390463 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.393490 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:47:12.393494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:47:12.393496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:47:12.393523 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:47:12.395320 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:12.395921 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:12.395924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:12.395926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:12.396007 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:12.396410 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:12.396413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:12.396415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:12.396630 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:47:12.396710 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:12.397361 (+ 651us) server_negotiation.cc:299] Negotiation successful
0923 14:47:12.397482 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:47:12.399441 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.382363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:43985 (local address 127.30.2.195:38069)
0923 14:47:12.383985 (+ 1622us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:12.384044 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:12.385447 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:12.386274 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:12.386283 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:12.386635 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:47:12.387350 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.387371 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.388483 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.388486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.389877 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.389889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.390664 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.390668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.390868 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.391546 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:47:12.391566 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:47:12.393321 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:47:12.395458 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:12.395463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:12.395479 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:12.395808 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:12.396124 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:12.396127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:12.396129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:12.396289 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:12.396774 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:47:12.396780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:47:12.397094 (+ 314us) client_negotiation.cc:769] Sending connection context
0923 14:47:12.398073 (+ 979us) client_negotiation.cc:240] Negotiation successful
0923 14:47:12.398377 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"thread_start_us":135,"threads_started":1}
I20260923 14:47:12.400941 10007 heartbeater.cc:347] Connected to a master server at 127.30.2.254:43985
I20260923 14:47:12.401185 10007 heartbeater.cc:464] Registering TS with master...
I20260923 14:47:12.401726 10007 heartbeater.cc:511] Master 127.30.2.254:43985 requested a full tablet report, sending...
I20260923 14:47:12.403017 9523 ts_manager.cc:194] Registered new tserver with Master: bf787c48a3bf4483a70f8a1f9e73713c (127.30.2.195:37539)
I20260923 14:47:12.403740 9523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:38069
I20260923 14:47:12.405822 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:47:12.422814 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.408872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.30.2.254:43985)
0923 14:47:12.408984 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:12.408988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:12.411294 (+ 2306us) server_negotiation.cc:407] Connection header received
0923 14:47:12.411488 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:12.411491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:12.411554 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:12.411609 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:47:12.412640 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.413658 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.415584 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.415956 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.416035 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.418233 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:47:12.418237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:47:12.418240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:47:12.418266 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:47:12.420087 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:12.421232 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:12.421236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:12.421238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:12.421321 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:12.421633 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:12.421637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:12.421639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:12.421846 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:47:12.421948 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:12.422487 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 14:47:12.422619 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 14:47:12 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174831, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:12.427038 9523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:47:12.440944 9523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:47:12.465643 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.454681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:34055 (local address 127.0.0.1:52962)
0923 14:47:12.457971 (+ 3290us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:12.457985 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:12.458072 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:12.460198 (+ 2126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:12.460202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:12.460220 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:47:12.460347 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.460353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.461983 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.461986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.463215 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.463226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.465049 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.465056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.465168 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.465203 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 14:47:12.465309 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 14:47:12.465465 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3076,"thread_start_us":111,"threads_started":1}
I20260923 14:47:12.466766 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.455451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.30.2.194:34055)
0923 14:47:12.458579 (+ 3128us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:12.458593 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:12.458614 (+ 21us) server_negotiation.cc:407] Connection header received
0923 14:47:12.458853 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:12.458858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:12.459904 (+ 1046us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:12.460058 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:47:12.460479 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.461855 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.463366 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.464722 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.464813 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.464965 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:12.466479 (+ 1514us) server_negotiation.cc:299] Negotiation successful
0923 14:47:12.466586 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3027,"thread_start_us":106,"threads_started":1}
I20260923 14:47:12.467002 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.456773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.30.2.195:37539)
0923 14:47:12.457676 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:12.457689 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:12.458336 (+ 647us) server_negotiation.cc:407] Connection header received
0923 14:47:12.458452 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:12.458457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:12.458653 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:12.460019 (+ 1366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:47:12.460156 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.461363 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.463861 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.464658 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.465093 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.465234 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:12.465715 (+ 481us) server_negotiation.cc:299] Negotiation successful
0923 14:47:12.466810 (+ 1095us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":94,"threads_started":1}
I20260923 14:47:12.467259 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.456353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:37539 (local address 127.0.0.1:50844)
0923 14:47:12.458257 (+ 1904us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:12.458269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:12.458346 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:12.459155 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:12.459158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:12.459173 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:47:12.459284 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.459292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.461494 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.461497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.463605 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.463614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.465293 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.465300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.465465 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.465485 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:47:12.465567 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 14:47:12.465623 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1835,"thread_start_us":78,"threads_started":1}
I20260923 14:47:12.470511 9806 tablet_service.cc:1467] Processing CreateTablet for tablet 8b96bb1080c64392a93f587966f0987d (DEFAULT_TABLE table=test-table [id=6cfbd0cbf6934490b9f7ff63e32cbcb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:47:12.470466 9929 tablet_service.cc:1467] Processing CreateTablet for tablet 8b96bb1080c64392a93f587966f0987d (DEFAULT_TABLE table=test-table [id=6cfbd0cbf6934490b9f7ff63e32cbcb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:47:12.471743 9929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b96bb1080c64392a93f587966f0987d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:47:12.471743 9806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b96bb1080c64392a93f587966f0987d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:47:12.480856 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.454624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:35589 (local address 127.0.0.1:38550)
0923 14:47:12.456080 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:12.456127 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:12.456255 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:12.457948 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:12.457955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:12.457982 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:47:12.458172 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.458180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.472733 (+ 14553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.472737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.473848 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.473859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.480368 (+ 6509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.480373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.480490 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.480536 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 14:47:12.480594 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 14:47:12.480681 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"thread_start_us":106,"threads_started":1}
I20260923 14:47:12.481962 10022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.455470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.30.2.193:35589)
0923 14:47:12.456887 (+ 1417us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:12.456895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:12.456918 (+ 23us) server_negotiation.cc:407] Connection header received
0923 14:47:12.456994 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:12.456999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:12.457535 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:12.458726 (+ 1191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:47:12.459100 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.469182 (+ 10082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.478338 (+ 9156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.479372 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.481469 (+ 2097us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.481648 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:12.481703 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 14:47:12.481801 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1308,"thread_start_us":90,"threads_started":1}
I20260923 14:47:12.482707 10025 tablet_bootstrap.cc:492] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c: Bootstrap starting.
I20260923 14:47:12.485980 10025 tablet_bootstrap.cc:654] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c: Neither blocks nor log segments found. Creating new log.
I20260923 14:47:12.486146 9669 tablet_service.cc:1467] Processing CreateTablet for tablet 8b96bb1080c64392a93f587966f0987d (DEFAULT_TABLE table=test-table [id=6cfbd0cbf6934490b9f7ff63e32cbcb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:47:12.486886 10025 log.cc:824] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c: Log is configured to *not* fsync() on all Append() calls
I20260923 14:47:12.488595 9669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b96bb1080c64392a93f587966f0987d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:47:12.489240 10025 tablet_bootstrap.cc:492] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c: No bootstrap required, opened a new log
I20260923 14:47:12.489436 10025 ts_tablet_manager.cc:1397] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260923 14:47:12.494059 10025 raft_consensus.cc:348] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea97e0c85354ead8db9cfe554154004" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35589 } } peers { permanent_uuid: "fd9228f437d2462eace71a6f2bbebda7" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 34055 } } peers { permanent_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 37539 } }
I20260923 14:47:12.494340 10025 raft_consensus.cc:374] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:47:12.494395 10025 raft_consensus.cc:729] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf787c48a3bf4483a70f8a1f9e73713c, State: Initialized, Role: FOLLOWER
I20260923 14:47:12.494912 10025 consensus_queue.cc:260] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea97e0c85354ead8db9cfe554154004" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35589 } } peers { permanent_uuid: "fd9228f437d2462eace71a6f2bbebda7" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 34055 } } peers { permanent_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 37539 } }
I20260923 14:47:12.496004 10025 ts_tablet_manager.cc:1428] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c: Time spent starting tablet: real 0.007s user 0.001s sys 0.007s
I20260923 14:47:12.496380 10007 heartbeater.cc:503] Master 127.30.2.254:43985 was elected leader, sending a full tablet report...
I20260923 14:47:12.503495 10029 tablet_bootstrap.cc:492] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: Bootstrap starting.
I20260923 14:47:12.506021 10029 tablet_bootstrap.cc:654] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: Neither blocks nor log segments found. Creating new log.
I20260923 14:47:12.507510 10029 log.cc:824] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: Log is configured to *not* fsync() on all Append() calls
I20260923 14:47:12.508328 10028 tablet_bootstrap.cc:492] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004: Bootstrap starting.
I20260923 14:47:12.511025 10028 tablet_bootstrap.cc:654] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004: Neither blocks nor log segments found. Creating new log.
I20260923 14:47:12.512199 10028 log.cc:824] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004: Log is configured to *not* fsync() on all Append() calls
I20260923 14:47:12.513763 10029 tablet_bootstrap.cc:492] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: No bootstrap required, opened a new log
I20260923 14:47:12.513986 10029 ts_tablet_manager.cc:1397] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260923 14:47:12.514345 10028 tablet_bootstrap.cc:492] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004: No bootstrap required, opened a new log
I20260923 14:47:12.514549 10028 ts_tablet_manager.cc:1397] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:47:12.518146 10029 raft_consensus.cc:348] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea97e0c85354ead8db9cfe554154004" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35589 } } peers { permanent_uuid: "fd9228f437d2462eace71a6f2bbebda7" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 34055 } } peers { permanent_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 37539 } }
I20260923 14:47:12.518424 10029 raft_consensus.cc:374] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:47:12.518468 10029 raft_consensus.cc:729] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9228f437d2462eace71a6f2bbebda7, State: Initialized, Role: FOLLOWER
I20260923 14:47:12.518994 10029 consensus_queue.cc:260] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea97e0c85354ead8db9cfe554154004" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35589 } } peers { permanent_uuid: "fd9228f437d2462eace71a6f2bbebda7" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 34055 } } peers { permanent_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 37539 } }
I20260923 14:47:12.519261 10028 raft_consensus.cc:348] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea97e0c85354ead8db9cfe554154004" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35589 } } peers { permanent_uuid: "fd9228f437d2462eace71a6f2bbebda7" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 34055 } } peers { permanent_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 37539 } }
I20260923 14:47:12.519485 10028 raft_consensus.cc:374] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:47:12.519526 10028 raft_consensus.cc:729] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea97e0c85354ead8db9cfe554154004, State: Initialized, Role: FOLLOWER
I20260923 14:47:12.520222 10029 ts_tablet_manager.cc:1428] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 14:47:12.520187 10028 consensus_queue.cc:260] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea97e0c85354ead8db9cfe554154004" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35589 } } peers { permanent_uuid: "fd9228f437d2462eace71a6f2bbebda7" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 34055 } } peers { permanent_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 37539 } }
I20260923 14:47:12.520614 9873 heartbeater.cc:503] Master 127.30.2.254:43985 was elected leader, sending a full tablet report...
I20260923 14:47:12.521214 10028 ts_tablet_manager.cc:1428] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 14:47:12.521513 9735 heartbeater.cc:503] Master 127.30.2.254:43985 was elected leader, sending a full tablet report...
W20260923 14:47:12.629112 10008 tablet.cc:2369] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:47:12.634723 9736 tablet.cc:2369] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:47:12.640415 10033 raft_consensus.cc:482] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:47:12.640584 10033 raft_consensus.cc:504] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea97e0c85354ead8db9cfe554154004" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35589 } } peers { permanent_uuid: "fd9228f437d2462eace71a6f2bbebda7" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 34055 } } peers { permanent_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 37539 } }
I20260923 14:47:12.641956 10033 leader_election.cc:288] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd9228f437d2462eace71a6f2bbebda7 (127.30.2.194:34055), bf787c48a3bf4483a70f8a1f9e73713c (127.30.2.195:37539)
I20260923 14:47:12.650528 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.643489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:42551 (local address 127.30.2.194:34055)
0923 14:47:12.643629 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:12.643634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:12.644538 (+ 904us) server_negotiation.cc:407] Connection header received
0923 14:47:12.644778 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:12.644782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:12.644833 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:12.644882 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:47:12.645368 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.646176 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.649196 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.649785 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.649895 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.649952 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:12.650312 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 14:47:12.650390 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:47:12.650897 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.643736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:50677 (local address 127.30.2.195:37539)
0923 14:47:12.643873 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:12.643877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:12.646839 (+ 2962us) server_negotiation.cc:407] Connection header received
0923 14:47:12.647016 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:12.647019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:12.647066 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:12.647107 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:47:12.647438 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.648171 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.649694 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.650256 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.650344 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.650391 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:12.650689 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 14:47:12.650784 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 14:47:12.651108 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.642598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:34055 (local address 127.30.2.193:42551)
0923 14:47:12.644430 (+ 1832us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:12.644447 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:12.644642 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:12.645042 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:12.645045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:12.645064 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:47:12.645177 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.645185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.647705 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.647709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.649030 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.649041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.650064 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.650082 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.650168 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.650193 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 14:47:12.650805 (+ 612us) client_negotiation.cc:240] Negotiation successful
0923 14:47:12.650882 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1719,"thread_start_us":149,"threads_started":1}
I20260923 14:47:12.651875 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.643052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:37539 (local address 127.30.2.193:50677)
0923 14:47:12.646721 (+ 3669us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:12.646760 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:12.646908 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:12.647236 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:12.647238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:12.647252 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:47:12.647332 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.647337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.648357 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.648360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.649551 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:12.649562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.650480 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.650483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:12.650566 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.650585 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 14:47:12.651660 (+ 1075us) client_negotiation.cc:240] Negotiation successful
0923 14:47:12.651729 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3603,"thread_start_us":108,"threads_started":1}
I20260923 14:47:12.651960 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96bb1080c64392a93f587966f0987d" candidate_uuid: "0ea97e0c85354ead8db9cfe554154004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9228f437d2462eace71a6f2bbebda7" is_pre_election: true
I20260923 14:47:12.652314 9811 raft_consensus.cc:2454] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea97e0c85354ead8db9cfe554154004 in term 0.
I20260923 14:47:12.652511 9961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96bb1080c64392a93f587966f0987d" candidate_uuid: "0ea97e0c85354ead8db9cfe554154004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" is_pre_election: true
I20260923 14:47:12.652805 9961 raft_consensus.cc:2454] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea97e0c85354ead8db9cfe554154004 in term 0.
I20260923 14:47:12.652837 9623 leader_election.cc:302] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea97e0c85354ead8db9cfe554154004, fd9228f437d2462eace71a6f2bbebda7; no voters:
I20260923 14:47:12.653131 10033 raft_consensus.cc:2781] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:47:12.653213 10033 raft_consensus.cc:482] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:47:12.653266 10033 raft_consensus.cc:3037] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:47:12.654697 10033 raft_consensus.cc:504] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea97e0c85354ead8db9cfe554154004" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35589 } } peers { permanent_uuid: "fd9228f437d2462eace71a6f2bbebda7" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 34055 } } peers { permanent_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 37539 } }
I20260923 14:47:12.655097 10033 leader_election.cc:288] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [CANDIDATE]: Term 1 election: Requested vote from peers fd9228f437d2462eace71a6f2bbebda7 (127.30.2.194:34055), bf787c48a3bf4483a70f8a1f9e73713c (127.30.2.195:37539)
I20260923 14:47:12.655406 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96bb1080c64392a93f587966f0987d" candidate_uuid: "0ea97e0c85354ead8db9cfe554154004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9228f437d2462eace71a6f2bbebda7"
I20260923 14:47:12.655527 9961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96bb1080c64392a93f587966f0987d" candidate_uuid: "0ea97e0c85354ead8db9cfe554154004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf787c48a3bf4483a70f8a1f9e73713c"
I20260923 14:47:12.655541 9811 raft_consensus.cc:3037] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:47:12.655629 9961 raft_consensus.cc:3037] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:47:12.656818 9811 raft_consensus.cc:2454] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea97e0c85354ead8db9cfe554154004 in term 1.
I20260923 14:47:12.656818 9961 raft_consensus.cc:2454] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea97e0c85354ead8db9cfe554154004 in term 1.
I20260923 14:47:12.657133 9623 leader_election.cc:302] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea97e0c85354ead8db9cfe554154004, bf787c48a3bf4483a70f8a1f9e73713c; no voters:
I20260923 14:47:12.657294 10033 raft_consensus.cc:2781] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:47:12.657545 10033 raft_consensus.cc:686] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [term 1 LEADER]: Becoming Leader. State: Replica: 0ea97e0c85354ead8db9cfe554154004, State: Running, Role: LEADER
I20260923 14:47:12.657853 10033 consensus_queue.cc:237] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea97e0c85354ead8db9cfe554154004" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35589 } } peers { permanent_uuid: "fd9228f437d2462eace71a6f2bbebda7" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 34055 } } peers { permanent_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 37539 } }
I20260923 14:47:12.660856 9523 catalog_manager.cc:5495] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ea97e0c85354ead8db9cfe554154004 (127.30.2.193). New cstate: current_term: 1 leader_uuid: "0ea97e0c85354ead8db9cfe554154004" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea97e0c85354ead8db9cfe554154004" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd9228f437d2462eace71a6f2bbebda7" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 37539 } health_report { overall_health: UNKNOWN } } }
I20260923 14:47:12.708613 10022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.703106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.30.2.193:35589)
0923 14:47:12.703250 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:12.703255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:12.703660 (+ 405us) server_negotiation.cc:407] Connection header received
0923 14:47:12.703901 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:12.703904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:12.703967 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:12.704091 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:47:12.704353 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.705028 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.706257 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:12.706554 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:12.706697 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:12.707087 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:47:12.707731 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:47:12.707948 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:12.708379 (+ 431us) server_negotiation.cc:299] Negotiation successful
0923 14:47:12.708444 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:47:12.719060 9961 raft_consensus.cc:1261] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c [term 1 FOLLOWER]: Refusing update from remote peer 0ea97e0c85354ead8db9cfe554154004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:47:12.719060 9811 raft_consensus.cc:1261] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7 [term 1 FOLLOWER]: Refusing update from remote peer 0ea97e0c85354ead8db9cfe554154004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:47:12.720422 10036 consensus_queue.cc:1035] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf787c48a3bf4483a70f8a1f9e73713c" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 37539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 23 14:47:12 dist-test-slave-wjf2 krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174831, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:12.720934 10036 consensus_queue.cc:1035] T 8b96bb1080c64392a93f587966f0987d P 0ea97e0c85354ead8db9cfe554154004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9228f437d2462eace71a6f2bbebda7" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 14:47:12.729068 9874 tablet.cc:2369] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:47:12.738080 10040 mvcc.cc:205] Tried to move back new op lower bound from 7332556114807508992 to 7332556114569072640. Current Snapshot: MvccSnapshot[applied={T|T < 7332556114807508992}]
I20260923 14:47:12.746469 10041 mvcc.cc:205] Tried to move back new op lower bound from 7332556114807508992 to 7332556114569072640. Current Snapshot: MvccSnapshot[applied={T|T < 7332556114807508992}]
I20260923 14:47:12.748179 9522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:47:12.748354 9522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:47:12.751099 9522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d448b253efd43e68b6cbe18913af535: Sending DeleteTablet for 3 replicas of tablet 8b96bb1080c64392a93f587966f0987d
I20260923 14:47:12.752008 9669 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b96bb1080c64392a93f587966f0987d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:47:12 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:38550
I20260923 14:47:12.752008 9806 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b96bb1080c64392a93f587966f0987d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:47:12 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:52962
I20260923 14:47:12.752729 9929 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b96bb1080c64392a93f587966f0987d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:47:12 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:50844
I20260923 14:47:12.753777 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 9604
I20260923 14:47:12.757089 10042 mvcc.cc:205] Tried to move back new op lower bound from 7332556114807508992 to 7332556114569072640. Current Snapshot: MvccSnapshot[applied={T|T < 7332556114807508992}]
I20260923 14:47:12.757428 10049 tablet_replica.cc:330] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: stopping tablet replica
I20260923 14:47:12.757754 10049 raft_consensus.cc:2229] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:47:12.758080 10049 raft_consensus.cc:2258] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:47:12.761169 10049 ts_tablet_manager.cc:1905] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 14:47:12.764482 9507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:35589 (error 108)
I20260923 14:47:12.764587 10049 ts_tablet_manager.cc:1918] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:47:12.764667 10049 log.cc:1196] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/8b96bb1080c64392a93f587966f0987d
I20260923 14:47:12.764983 10049 ts_tablet_manager.cc:1939] T 8b96bb1080c64392a93f587966f0987d P fd9228f437d2462eace71a6f2bbebda7: Deleting consensus metadata
I20260923 14:47:12.766083 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:12.765766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:35589 (local address 127.0.0.1:38560)
0923 14:47:12.765933 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:12.765996 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260923 14:47:12.766441 9508 catalog_manager.cc:4841] TS fd9228f437d2462eace71a6f2bbebda7 (127.30.2.194:34055): tablet 8b96bb1080c64392a93f587966f0987d (table test-table [id=6cfbd0cbf6934490b9f7ff63e32cbcb3]) successfully deleted
W20260923 14:47:12.766716 9507 catalog_manager.cc:4584] TS 0ea97e0c85354ead8db9cfe554154004 (127.30.2.193:35589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b96bb1080c64392a93f587966f0987d: Network error: Client connection negotiation failed: client connection to 127.30.2.193:35589: connect: Connection refused (error 111)
I20260923 14:47:12.767096 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 9743
I20260923 14:47:12.770632 10050 tablet_replica.cc:330] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c: stopping tablet replica
I20260923 14:47:12.771054 10050 raft_consensus.cc:2229] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:47:12.771319 10050 raft_consensus.cc:2258] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:47:12.774821 10050 ts_tablet_manager.cc:1905] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:47:12.778345 10050 ts_tablet_manager.cc:1918] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:47:12.778498 10050 log.cc:1196] T 8b96bb1080c64392a93f587966f0987d P bf787c48a3bf4483a70f8a1f9e73713c: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/8b96bb1080c64392a93f587966f0987d
I20260923 14:47:12.779103 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 9878
W20260923 14:47:12.788791 10020 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:47:12.788120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:37539 (local address 127.0.0.1:50860)
0923 14:47:12.788319 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:12.788338 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:12.788443 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:12.788656 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.195:37539: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":113}
I20260923 14:47:12.789491 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 9491
I20260923 14:47:12.799649 30731 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 14:47:13.144749 30731 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 14:47:13.144865 30731 mini_ranger.cc:66] Stopping Ranger...
I20260923 14:47:13.489619 30731 mini_ranger.cc:68] Stopped Ranger
2026-09-23 14:47:13.489 UTC [9076] LOG: received smart shutdown request
2026-09-23 14:47:13.493 UTC [9076] LOG: background worker "logical replication launcher" (PID 9085) exited with exit code 1
2026-09-23 14:47:13.494 UTC [9080] LOG: shutting down
2026-09-23 14:47:13.694 UTC [9076] LOG: database system is shut down
2026-09-23T14:47:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48840 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:47:13 dist-test-slave-wjf2 krb5kdc[10062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:47:13 dist-test-slave-wjf2 krb5kdc[10062](info): set up 2 sockets
Sep 23 14:47:13 dist-test-slave-wjf2 krb5kdc[10062](info): commencing operation
krb5kdc: starting...
W20260923 14:47:14.893178 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:47:14 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:47:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/rangeradmin_127.30.2.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/rangeradmin_127.30.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/rangerlookup_127.30.2.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/rangerlookup_127.30.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/HTTP_127.30.2.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/HTTP_127.30.2.212@KRBTEST.COM.keytab.
I20260923 14:47:15.075042 30731 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknhhgz8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/postgres -l logfile start
2026-09-23 14:47:17.752 UTC [10092] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 14:47:17.752 UTC [10092] LOG: listening on IPv4 address "127.30.2.212", port 54935
2026-09-23 14:47:17.756 UTC [10092] LOG: listening on Unix socket "/tmp/.s.PGSQL.54935"
2026-09-23 14:47:17.794 UTC [10095] LOG: database system was shut down at 2026-09-23 14:47:16 UTC
2026-09-23 14:47:17.798 UTC [10092] LOG: database system is ready to accept connections
I20260923 14:47:18.849006 30731 mini_postgres.cc:96] Postgres bound to 54935
2026-09-23 14:47:18.855 UTC [10103] FATAL: database "slave" does not exist
127.30.2.212:54935 - accepting connections
I20260923 14:47:18.856106 30731 mini_ranger.cc:161] Starting Ranger...
I20260923 14:47:18.871158 30731 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 14:47:19.424767 30731 mini_ranger.cc:87] Created ranger Postgres database
I20260923 14:47:19.424857 30731 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-admin
2026-09-23 14:47:19,698 [I] DB FLAVOR :POSTGRES
2026-09-23 14:47:19,698 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 14:47:19,698 [I] Checking connection..
2026-09-23 14:47:19,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 14:47:19,934 [I] Checking connection passed.
2026-09-23 14:47:19,934 [I] --------- Verifying version history table ---------
2026-09-23 14:47:19,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:47:20,168 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 14:47:20,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:47:20,412 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 14:47:20,412 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 14:47:20,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 14:47:20.666 UTC [10167] WARNING: there is no transaction in progress
2026-09-23 14:47:20,672 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 14:47:20,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:47:20,927 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 14:47:20,927 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 14:47:20,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 14:47:21,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 14:47:21,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wjf2.c.gcp-upstream.internal','N') ;"
2026-09-23 14:47:21,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 14:47:21,906 [I] Table x_portal_user does not exist in database ranger
2026-09-23 14:47:21,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 14:47:22,158 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 14:47:22,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 14:47:22,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 14:47:22,647 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 14:47:22,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 14:47:23.678 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.694 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.712 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.729 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.746 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.765 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.820 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.826 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.842 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.852 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.862 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.877 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.888 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.895 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.903 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:23.911 UTC [10275] WARNING: there is no transaction in progress
2026-09-23 14:47:24,566 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 14:47:24,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 14:47:24,791 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 14:47:24,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 14:47:25,050 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 14:47:25,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 14:47:25,291 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 14:47:25,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 14:47:25,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 14:47:25,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wjf2.c.gcp-upstream.internal';"
2026-09-23 14:47:26,050 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 14:47:26,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 14:47:26,297 [I] DB_PATCHES have already been applied
I20260923 14:47:26.302718 30731 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknhhgz8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 14:47:26.302774 30731 mini_ranger.cc:193] Using host: 127.30.2.212
I20260923 14:47:26.306262 30731 mini_ranger.cc:227] Ranger admin URL: http://127.30.2.212:55393
Sep 23, 2026 2:47:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 2:47:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 2:47:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 2:47:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 2:47:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 2:47:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 2:47:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 14:47:27.434357 30731 mini_ranger.cc:160] Time spent starting Ranger: real 8.578s user 0.004s sys 0.006s
Sep 23, 2026 2:47:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 2:47:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 23, 2026 2:47:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 2:47:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 2:47:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 14:47:49.033591 30731 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/rangerkms_127.30.2.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/rangerkms_127.30.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.2.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.2.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.2.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/HTTP_127.30.2.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.2.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/HTTP_127.30.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 14:47:49 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174869, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 14:47:49.141701 30731 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 14:47:49.152638 30731 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 14:47:50.083948 30731 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 14:47:50.084038 30731 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-kms
I20260923 14:47:50.084059 30731 mini_ranger_kms.cc:227] Using postgres at 127.30.2.212:54935
2026-09-23 14:47:50,254 [I] DB FLAVOR :POSTGRES
2026-09-23 14:47:50,255 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 14:47:50,255 [I] Checking connection
2026-09-23 14:47:50,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 14:47:50,497 [I] connection success
2026-09-23 14:47:50,497 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 14:47:50,497 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 14:47:50,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 14:47:50,725 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 14:47:50,726 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 14:47:50,726 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 14:47:50,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.2.212:54935/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 14:47:50,997 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 14:47:51.433960 30731 mini_ranger_kms.cc:310] Created kms service
I20260923 14:47:51.641712 30731 mini_ranger_kms.cc:326] Created kudu user
I20260923 14:47:51.685779 30731 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 14:47:51 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174869, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:52.095376 30731 mini_ranger_kms.cc:384] Added ranger policy
I20260923 14:47:52.095535 30731 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknhhgz8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknhhgz8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 14:47:52.095557 30731 mini_ranger_kms.cc:240] Using host: 127.30.2.212
I20260923 14:47:52.099363 30731 mini_ranger_kms.cc:276] Ranger KMS PID: 10456
I20260923 14:47:52.099454 30731 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.2.212:54251
Sep 23, 2026 2:47:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 2:47:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 2:47:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 2:47:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 2:47:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 2:47:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 14:47:54.342244 30731 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.201s user 0.012s sys 0.142s
I20260923 14:47:54.342361 30731 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 14:47:54.342420 30731 mini_ranger_kms.cc:401] 127.30.2.212:54251/kms/v1/keys
Sep 23 14:47:54 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790174874, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.2.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 14:47:54 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174874, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.2.212@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
Sep 23 14:47:55 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174874, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.2.212@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
Sep 23 14:47:57 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:47:57.384102 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:42455
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:42455
--ranger_config_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:54251/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:47:57.538025 10507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:47:57.538276 10507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:47:57.538314 10507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:47:57.542138 10507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:47:57.542183 10507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:47:57.542202 10507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:47:57.542218 10507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:47:57.542232 10507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:47:57.547418 10507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.2.212:54251/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:42455
--ranger_config_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:42455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:47:57.548337 10507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:47:57.549253 10507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:47:57.557739 10515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:57.557740 10512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:57.557796 10507 server_base.cc:1034] running on GCE node
W20260923 14:47:57.557724 10513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:57.558584 10507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:47:57.559656 10507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:47:57.560811 10507 hybrid_clock.cc:648] HybridClock initialized: now 1790174877560798 us; error 24 us; skew 500 ppm
Sep 23 14:47:57 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174877, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:47:57.563391 10507 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:47:57.564257 10507 webserver.cc:460] Webserver started at http://127.30.2.254:44897/ using document root <none> and password file <none>
I20260923 14:47:57.564754 10507 fs_manager.cc:359] Metadata directory not provided
I20260923 14:47:57.564805 10507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:47:57.564962 10507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:47:57 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174877, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:57.610482 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "8788a18b41914418adecf5cf0b16cc87"
format_stamp: "Formatted at 2026-09-23 14:47:57 on dist-test-slave-wjf2"
server_key: "e64a398fd9eda9d28fd865aa86ca2a39"
server_key_iv: "158da3e61bb1b3ee16014a579630b383"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:57.611128 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "8788a18b41914418adecf5cf0b16cc87"
format_stamp: "Formatted at 2026-09-23 14:47:57 on dist-test-slave-wjf2"
server_key: "e64a398fd9eda9d28fd865aa86ca2a39"
server_key_iv: "158da3e61bb1b3ee16014a579630b383"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:57.615342 10507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 14:47:57.629349 10522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:57.630834 10507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 14:47:57.630981 10507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "8788a18b41914418adecf5cf0b16cc87"
format_stamp: "Formatted at 2026-09-23 14:47:57 on dist-test-slave-wjf2"
server_key: "e64a398fd9eda9d28fd865aa86ca2a39"
server_key_iv: "158da3e61bb1b3ee16014a579630b383"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:57.631115 10507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:47:57.644392 10507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:47:57.647415 10507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:47:57.647619 10507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:47:57.655675 10507 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:42455
I20260923 14:47:57.655682 10574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:42455 every 8 connection(s)
I20260923 14:47:57.656630 10507 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:47:57.659672 10575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:47:57.662484 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 10507
I20260923 14:47:57.662628 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:47:57.662966 30731 external_mini_cluster.cc:1428] Setting key cc6013a5f3c783f8a5f24f80ace00013
I20260923 14:47:57.666046 10575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87: Bootstrap starting.
I20260923 14:47:57.668706 10575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87: Neither blocks nor log segments found. Creating new log.
I20260923 14:47:57.669534 10575 log.cc:824] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87: Log is configured to *not* fsync() on all Append() calls
I20260923 14:47:57.671741 10575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87: No bootstrap required, opened a new log
Sep 23 14:47:57 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:47:57.676148 10575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } }
I20260923 14:47:57.676433 10575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:47:57.676522 10575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8788a18b41914418adecf5cf0b16cc87, State: Initialized, Role: FOLLOWER
I20260923 14:47:57.677052 10575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } }
I20260923 14:47:57.677238 10575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:47:57.677326 10575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:47:57.677446 10575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:47:57.678339 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:57.664453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.30.2.254:42455)
0923 14:47:57.664927 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:57.664947 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:57.664992 (+ 45us) server_negotiation.cc:407] Connection header received
0923 14:47:57.665868 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:57.665895 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:57.666334 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:57.666771 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:47:57.667613 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:57.669206 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:57.670293 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:57.670771 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:57.670944 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:57.673504 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:47:57.673510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:47:57.673523 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:47:57.673549 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:47:57.675167 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:57.675630 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:57.675635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:57.675639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:57.675755 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:57.676087 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:57.676090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:57.676092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:57.676545 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:47:57.676726 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:57.677173 (+ 447us) server_negotiation.cc:299] Negotiation successful
0923 14:47:57.677635 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":132,"threads_started":1}
I20260923 14:47:57.678555 10575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } }
I20260923 14:47:57.678977 10575 leader_election.cc:302] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8788a18b41914418adecf5cf0b16cc87; no voters:
I20260923 14:47:57.679340 10575 leader_election.cc:288] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:47:57.679504 10580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:47:57.679755 10580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 1 LEADER]: Becoming Leader. State: Replica: 8788a18b41914418adecf5cf0b16cc87, State: Running, Role: LEADER
I20260923 14:47:57.680145 10580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } }
I20260923 14:47:57.680740 10575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:47:57.681429 10580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8788a18b41914418adecf5cf0b16cc87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } } }
I20260923 14:47:57.681555 10580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [sys.catalog]: This master's current role is: LEADER
I20260923 14:47:57.681797 10580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8788a18b41914418adecf5cf0b16cc87. Latest consensus state: current_term: 1 leader_uuid: "8788a18b41914418adecf5cf0b16cc87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } } }
I20260923 14:47:57.681910 10580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [sys.catalog]: This master's current role is: LEADER
I20260923 14:47:57.682154 10587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:47:57.686026 10587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:47:57.686452 10575 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 14:47:57.696738 10587 catalog_manager.cc:1347] Generated new cluster ID: 25b24e87e1054ae5a2a04733b75365a4
I20260923 14:47:57.696815 10587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:47:57.730937 10587 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:47:57.732456 10587 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:47:57.747869 10587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87: Generated new TSK 0
I20260923 14:47:57.748623 10587 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:47:58.378756 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:54251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42455
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:47:58.522919 10620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:47:58.523239 10620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:47:58.523375 10620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:47:58.529011 10620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:47:58.529155 10620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:47:58.529294 10620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:47:58.535550 10620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.2.212:54251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:47:58.536728 10620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:47:58.537810 10620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:47:58.547389 10627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:58.547400 10630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:58.547657 10628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:58.548550 10620 server_base.cc:1034] running on GCE node
I20260923 14:47:58.548962 10620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:47:58.549686 10620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:47:58.550915 10620 hybrid_clock.cc:648] HybridClock initialized: now 1790174878550885 us; error 44 us; skew 500 ppm
Sep 23 14:47:58 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174878, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:47:58.553988 10620 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:47:58.555126 10620 webserver.cc:460] Webserver started at http://127.30.2.193:42233/ using document root <none> and password file <none>
I20260923 14:47:58.555979 10620 fs_manager.cc:359] Metadata directory not provided
I20260923 14:47:58.556088 10620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:47:58.556370 10620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:47:58 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174878, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:58.571996 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "37ca786467604d079a1855bd1a5b1f02"
format_stamp: "Formatted at 2026-09-23 14:47:58 on dist-test-slave-wjf2"
server_key: "fcc245939d50ce5ed80fb6602d23aa60"
server_key_iv: "6204b9c32cdb73b45bc779d5ce80b9cd"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:58.573086 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "37ca786467604d079a1855bd1a5b1f02"
format_stamp: "Formatted at 2026-09-23 14:47:58 on dist-test-slave-wjf2"
server_key: "fcc245939d50ce5ed80fb6602d23aa60"
server_key_iv: "6204b9c32cdb73b45bc779d5ce80b9cd"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:58.581404 10620 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260923 14:47:58.604161 10637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:58.605674 10620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:47:58.605824 10620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "37ca786467604d079a1855bd1a5b1f02"
format_stamp: "Formatted at 2026-09-23 14:47:58 on dist-test-slave-wjf2"
server_key: "fcc245939d50ce5ed80fb6602d23aa60"
server_key_iv: "6204b9c32cdb73b45bc779d5ce80b9cd"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:58.605971 10620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:47:58.635751 10620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:47:58.639499 10620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:47:58.639750 10620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:47:58.640565 10620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:47:58.641688 10620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:47:58.641798 10620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:58.641927 10620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:47:58.642022 10620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:58.653086 10620 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:40283
I20260923 14:47:58.654335 10620 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:47:58.655074 10750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:40283 every 8 connection(s)
I20260923 14:47:58.659432 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 10620
I20260923 14:47:58.659560 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:47:58.659929 30731 external_mini_cluster.cc:1428] Setting key d6e86fb9b77ae474f2259c4a0709804a
Sep 23 14:47:58 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174878, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:47:58.674017 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:58.657780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:43295 (local address 127.30.2.254:42455)
0923 14:47:58.658118 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:58.658132 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:58.658909 (+ 777us) server_negotiation.cc:407] Connection header received
0923 14:47:58.660244 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:58.660248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:58.660311 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:58.660389 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:47:58.662115 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:58.663014 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:58.664964 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:58.665389 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:58.665496 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:58.669114 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:47:58.669119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:47:58.669123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:47:58.669157 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:47:58.671241 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:58.671840 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:58.671844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:58.671846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:58.671953 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:58.672411 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:58.672415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:58.672417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:58.672698 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:47:58.672802 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:58.673613 (+ 811us) server_negotiation.cc:299] Negotiation successful
0923 14:47:58.673763 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
I20260923 14:47:58.675927 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:58.658267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:42455 (local address 127.30.2.193:43295)
0923 14:47:58.658753 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:58.658807 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:58.659978 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:58.660840 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:58.660850 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:58.661198 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:47:58.661921 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:58.661949 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:58.663199 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:58.663206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:58.664781 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:58.664800 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:58.665731 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:58.665738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:58.666013 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:58.666849 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:47:58.666877 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:47:58.668874 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:47:58.671391 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:58.671396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:58.671408 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:58.671679 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:58.672081 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:58.672085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:58.672087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:58.672271 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:58.672907 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:47:58.672925 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:47:58.673281 (+ 356us) client_negotiation.cc:769] Sending connection context
0923 14:47:58.674269 (+ 988us) client_negotiation.cc:240] Negotiation successful
0923 14:47:58.674660 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":153,"threads_started":1}
I20260923 14:47:58.677687 10751 heartbeater.cc:347] Connected to a master server at 127.30.2.254:42455
I20260923 14:47:58.677961 10751 heartbeater.cc:464] Registering TS with master...
I20260923 14:47:58.678566 10751 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
I20260923 14:47:58.680672 10539 ts_manager.cc:194] Registered new tserver with Master: 37ca786467604d079a1855bd1a5b1f02 (127.30.2.193:40283)
I20260923 14:47:58.682364 10539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:43295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:47:58.698940 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:54251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42455
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:47:58 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790174878, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 14:47:58.827812 10759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:47:58.828155 10759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:47:58.828262 10759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:47:58.833027 10759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:47:58.833132 10759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:47:58.833257 10759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:47:58.838898 10759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.2.212:54251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:47:58.840072 10759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:47:58.841126 10759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:47:58.852041 10765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:58.852417 10759 server_base.cc:1034] running on GCE node
W20260923 14:47:58.852747 10766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:58.852191 10768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:58.853219 10759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:47:58.853952 10759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:47:58.855150 10759 hybrid_clock.cc:648] HybridClock initialized: now 1790174878855125 us; error 39 us; skew 500 ppm
Sep 23 14:47:58 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174878, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:47:58.858219 10759 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:47:58.859207 10759 webserver.cc:460] Webserver started at http://127.30.2.194:44303/ using document root <none> and password file <none>
I20260923 14:47:58.859894 10759 fs_manager.cc:359] Metadata directory not provided
I20260923 14:47:58.860023 10759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:47:58.860260 10759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:47:58 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174878, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:58.875295 10759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "0ece16c017e44d20bea64604b0e37ff8"
format_stamp: "Formatted at 2026-09-23 14:47:58 on dist-test-slave-wjf2"
server_key: "4ad666868e6d188177d289f408017a6d"
server_key_iv: "f811d82728f22554eb7f3f6d610c1d49"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:58.876075 10759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "0ece16c017e44d20bea64604b0e37ff8"
format_stamp: "Formatted at 2026-09-23 14:47:58 on dist-test-slave-wjf2"
server_key: "4ad666868e6d188177d289f408017a6d"
server_key_iv: "f811d82728f22554eb7f3f6d610c1d49"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:58.884863 10759 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260923 14:47:58.902721 10775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:58.904824 10759 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 14:47:58.904973 10759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "0ece16c017e44d20bea64604b0e37ff8"
format_stamp: "Formatted at 2026-09-23 14:47:58 on dist-test-slave-wjf2"
server_key: "4ad666868e6d188177d289f408017a6d"
server_key_iv: "f811d82728f22554eb7f3f6d610c1d49"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:58.905180 10759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:47:58.928992 10759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:47:58.932760 10759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:47:58.933007 10759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:47:58.933753 10759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:47:58.934845 10759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:47:58.934935 10759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:58.935050 10759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:47:58.935113 10759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:58.945664 10759 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:33945
I20260923 14:47:58.946900 10759 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:47:58.956166 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 10759
I20260923 14:47:58.956321 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:47:58.956723 30731 external_mini_cluster.cc:1428] Setting key 60fc4caca44732ab5df8a3de222b5047
I20260923 14:47:58.958153 10888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:33945 every 8 connection(s)
Sep 23 14:47:58 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174878, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:47:58.972843 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:58.956601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:48715 (local address 127.30.2.254:42455)
0923 14:47:58.956735 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:58.956739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:58.959158 (+ 2419us) server_negotiation.cc:407] Connection header received
0923 14:47:58.960427 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:58.960431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:58.960485 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:58.960538 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:47:58.962149 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:58.962980 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:58.964720 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:58.965085 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:58.965166 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:58.968405 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:47:58.968409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:47:58.968412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:47:58.968439 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:47:58.970283 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:58.970891 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:58.970895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:58.970897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:58.970996 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:58.971435 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:58.971440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:58.971441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:58.971687 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:47:58.971804 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:58.972512 (+ 708us) server_negotiation.cc:299] Negotiation successful
0923 14:47:58.972667 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:47:58.974750 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:58.957138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:42455 (local address 127.30.2.194:48715)
0923 14:47:58.959012 (+ 1874us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:58.959065 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:58.960180 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:58.960938 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:58.960964 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:58.961316 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:47:58.961976 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:58.961997 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:58.963145 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:58.963148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:58.964578 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:58.964590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:58.965391 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:58.965395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:58.965605 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:58.966323 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:47:58.966350 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:47:58.968217 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:47:58.970412 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:58.970418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:58.970434 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:58.970772 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:58.971109 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:58.971111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:58.971113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:58.971307 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:58.971893 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:47:58.971900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:47:58.972219 (+ 319us) client_negotiation.cc:769] Sending connection context
0923 14:47:58.973327 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0923 14:47:58.973678 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1643,"thread_start_us":156,"threads_started":1}
I20260923 14:47:58.976329 10889 heartbeater.cc:347] Connected to a master server at 127.30.2.254:42455
I20260923 14:47:58.976577 10889 heartbeater.cc:464] Registering TS with master...
I20260923 14:47:58.977197 10889 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
I20260923 14:47:58.978368 10539 ts_manager.cc:194] Registered new tserver with Master: 0ece16c017e44d20bea64604b0e37ff8 (127.30.2.194:33945)
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:47:58.979125 10539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:48715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:47:59.015185 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:54251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42455
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:47:59.174372 10894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:47:59.175077 10894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:47:59.175231 10894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:47:59.181552 10894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:47:59.181676 10894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:47:59.181808 10894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:47:59.189595 10894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.2.212:54251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:47:59.191147 10894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:47:59.192553 10894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:47:59.207844 10902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:59.207870 10900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:59.207919 10899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:59.209244 10894 server_base.cc:1034] running on GCE node
I20260923 14:47:59.209692 10894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:47:59.210443 10894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:47:59.211593 10894 hybrid_clock.cc:648] HybridClock initialized: now 1790174879211524 us; error 84 us; skew 500 ppm
Sep 23 14:47:59 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:47:59.215195 10894 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:47:59.216245 10894 webserver.cc:460] Webserver started at http://127.30.2.195:39689/ using document root <none> and password file <none>
I20260923 14:47:59.216905 10894 fs_manager.cc:359] Metadata directory not provided
I20260923 14:47:59.217022 10894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:47:59.217238 10894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:47:59 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:59.238607 10894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "a19b5e42b81d42c6819768cf43bb7ff0"
format_stamp: "Formatted at 2026-09-23 14:47:59 on dist-test-slave-wjf2"
server_key: "cbcd4fde02e93c135856ec992db48949"
server_key_iv: "68b7c35b17d3f0b0af064b9d59344245"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:59.252336 10894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "a19b5e42b81d42c6819768cf43bb7ff0"
format_stamp: "Formatted at 2026-09-23 14:47:59 on dist-test-slave-wjf2"
server_key: "cbcd4fde02e93c135856ec992db48949"
server_key_iv: "68b7c35b17d3f0b0af064b9d59344245"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:59.266010 10894 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.000s
I20260923 14:47:59.281333 10909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:59.283048 10894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 14:47:59.283171 10894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "a19b5e42b81d42c6819768cf43bb7ff0"
format_stamp: "Formatted at 2026-09-23 14:47:59 on dist-test-slave-wjf2"
server_key: "cbcd4fde02e93c135856ec992db48949"
server_key_iv: "68b7c35b17d3f0b0af064b9d59344245"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:59.283291 10894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:47:59.310752 10894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:47:59.314635 10894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:47:59.314878 10894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:47:59.315582 10894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:47:59.316672 10894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:47:59.316776 10894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:59.316871 10894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:47:59.316934 10894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:59.330049 10894 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:36215
I20260923 14:47:59.331318 10894 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:47:59.331987 11022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:36215 every 8 connection(s)
I20260923 14:47:59.338450 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 10894
I20260923 14:47:59.338588 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:47:59.338871 30731 external_mini_cluster.cc:1428] Setting key e1e765f428c31639727cc6b3079ea363
Sep 23 14:47:59 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:47:59.350087 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:59.335997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:41161 (local address 127.30.2.254:42455)
0923 14:47:59.336116 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:59.336121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:59.336738 (+ 617us) server_negotiation.cc:407] Connection header received
0923 14:47:59.337934 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:59.337938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:59.337992 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:59.338044 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:47:59.339894 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:59.340746 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:59.342402 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:59.342756 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:59.342826 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:59.345999 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:47:59.346003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:47:59.346006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:47:59.346032 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:47:59.347840 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:59.348421 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:59.348424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:59.348426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:59.348519 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:59.348878 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:59.348882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:59.348883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:59.349088 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:47:59.349167 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:59.349802 (+ 635us) server_negotiation.cc:299] Negotiation successful
0923 14:47:59.349925 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:47:59.351712 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:59.334908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:42455 (local address 127.30.2.195:41161)
0923 14:47:59.336590 (+ 1682us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:59.336642 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:59.337690 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:59.338465 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:59.338474 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:59.338814 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:47:59.339466 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:59.339483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:59.340900 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:59.340903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:59.342266 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:59.342277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:59.343018 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:59.343021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:59.343208 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:59.343903 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:47:59.343923 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:47:59.345812 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:47:59.347976 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:59.347982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:59.347998 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:59.348310 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:59.348621 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:59.348624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:59.348626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:59.348779 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:59.349230 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:47:59.349236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:47:59.349535 (+ 299us) client_negotiation.cc:769] Sending connection context
0923 14:47:59.350394 (+ 859us) client_negotiation.cc:240] Negotiation successful
0923 14:47:59.350723 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1453,"thread_start_us":141,"threads_started":1}
I20260923 14:47:59.353046 11023 heartbeater.cc:347] Connected to a master server at 127.30.2.254:42455
I20260923 14:47:59.353274 11023 heartbeater.cc:464] Registering TS with master...
I20260923 14:47:59.353811 11023 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
I20260923 14:47:59.354880 10538 ts_manager.cc:194] Registered new tserver with Master: a19b5e42b81d42c6819768cf43bb7ff0 (127.30.2.195:36215)
I20260923 14:47:59.355623 10538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:41161
Sep 23 14:47:59 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174878, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:59.364368 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:47:59.365185 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 10620
I20260923 14:47:59.374394 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 10759
I20260923 14:47:59.379895 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 10894
I20260923 14:47:59.387130 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 10507
I20260923 14:47:59.398808 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:42455
--webserver_interface=127.30.2.254
--webserver_port=44897
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:42455
--ranger_config_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:54251/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.2.212:54251/kms with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:47:59.519680 11026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:47:59.520036 11026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:47:59.520123 11026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:47:59.524204 11026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:47:59.524302 11026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:47:59.524351 11026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:47:59.524386 11026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:47:59.524418 11026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:47:59.529953 11026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.2.212:54251/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:42455
--ranger_config_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:42455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=44897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:47:59.530966 11026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:47:59.532100 11026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:47:59.540498 11034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:59.540526 11031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:59.540526 11032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:59.540845 11026 server_base.cc:1034] running on GCE node
I20260923 14:47:59.541507 11026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:47:59.542848 11026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:47:59.544029 11026 hybrid_clock.cc:648] HybridClock initialized: now 1790174879544005 us; error 34 us; skew 500 ppm
Sep 23 14:47:59 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:47:59.547142 11026 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:47:59.548095 11026 webserver.cc:460] Webserver started at http://127.30.2.254:44897/ using document root <none> and password file <none>
I20260923 14:47:59.548648 11026 fs_manager.cc:359] Metadata directory not provided
I20260923 14:47:59.548743 11026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 14:47:59 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:59.631855 11026 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:47:59.634306 11042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:59.635592 11026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:47:59.635743 11026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "8788a18b41914418adecf5cf0b16cc87"
format_stamp: "Formatted at 2026-09-23 14:47:57 on dist-test-slave-wjf2"
server_key: "e64a398fd9eda9d28fd865aa86ca2a39"
server_key_iv: "158da3e61bb1b3ee16014a579630b383"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:59.636380 11026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:47:59.650252 11026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:47:59.652907 11026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:47:59.653347 11026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:47:59.660441 11026 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:42455
I20260923 14:47:59.660456 11094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:42455 every 8 connection(s)
I20260923 14:47:59.661515 11026 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:47:59.665761 11095 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:47:59.667390 11095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87: Bootstrap starting.
I20260923 14:47:59.669124 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 11026
I20260923 14:47:59.669567 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:54251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:40283
--local_ip_for_outbound_sockets=127.30.2.193
--tserver_master_addrs=127.30.2.254:42455
--webserver_port=42233
--webserver_interface=127.30.2.193
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.2.212:54251/kms with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:47:59.683960 11095 log.cc:824] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87: Log is configured to *not* fsync() on all Append() calls
I20260923 14:47:59.690436 11095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:47:59.690821 11095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87: Bootstrap complete.
I20260923 14:47:59.694904 11095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } }
I20260923 14:47:59.695303 11095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8788a18b41914418adecf5cf0b16cc87, State: Initialized, Role: FOLLOWER
I20260923 14:47:59.695824 11095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } }
I20260923 14:47:59.695947 11095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:47:59.696028 11095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:47:59.696125 11095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:47:59.697896 11095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } }
I20260923 14:47:59.698221 11095 leader_election.cc:302] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8788a18b41914418adecf5cf0b16cc87; no voters:
I20260923 14:47:59.698550 11095 leader_election.cc:288] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 14:47:59.698622 11099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 14:47:59.698868 11099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [term 2 LEADER]: Becoming Leader. State: Replica: 8788a18b41914418adecf5cf0b16cc87, State: Running, Role: LEADER
I20260923 14:47:59.699209 11099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } }
I20260923 14:47:59.699739 11095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:47:59.700707 11101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8788a18b41914418adecf5cf0b16cc87. Latest consensus state: current_term: 2 leader_uuid: "8788a18b41914418adecf5cf0b16cc87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } } }
I20260923 14:47:59.700760 11100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8788a18b41914418adecf5cf0b16cc87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8788a18b41914418adecf5cf0b16cc87" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42455 } } }
I20260923 14:47:59.700846 11101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [sys.catalog]: This master's current role is: LEADER
I20260923 14:47:59.700871 11100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87 [sys.catalog]: This master's current role is: LEADER
I20260923 14:47:59.701373 11108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:47:59.704532 11108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:47:59.705049 11095 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 14:47:59.705228 11108 catalog_manager.cc:1259] Loaded cluster ID: 25b24e87e1054ae5a2a04733b75365a4
I20260923 14:47:59.705266 11108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:47:59.707257 11108 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:47:59.707912 11108 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87: Loaded TSK: 0
I20260923 14:47:59.711338 11108 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:47:59.813992 11097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:47:59.814385 11097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:47:59.814492 11097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:47:59.820261 11097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:47:59.820384 11097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:47:59.820528 11097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:47:59.828100 11097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.2.212:54251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:40283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=42233
--tserver_master_addrs=127.30.2.254:42455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:47:59.829449 11097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:47:59.830848 11097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:47:59.843530 11133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:59.843883 11097 server_base.cc:1034] running on GCE node
W20260923 14:47:59.843883 11135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:47:59.844020 11132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:47:59.844542 11097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:47:59.845397 11097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:47:59.848737 11097 hybrid_clock.cc:648] HybridClock initialized: now 1790174879848701 us; error 45 us; skew 500 ppm
Sep 23 14:47:59 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:47:59.852795 11097 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:47:59.853976 11097 webserver.cc:460] Webserver started at http://127.30.2.193:42233/ using document root <none> and password file <none>
I20260923 14:47:59.854746 11097 fs_manager.cc:359] Metadata directory not provided
I20260923 14:47:59.854887 11097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 14:47:59 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:47:59.888365 11097 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260923 14:47:59.891917 11143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:59.893635 11097 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 14:47:59.893771 11097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "37ca786467604d079a1855bd1a5b1f02"
format_stamp: "Formatted at 2026-09-23 14:47:58 on dist-test-slave-wjf2"
server_key: "fcc245939d50ce5ed80fb6602d23aa60"
server_key_iv: "6204b9c32cdb73b45bc779d5ce80b9cd"
server_key_version: "kuduclusterkey@0"
I20260923 14:47:59.894516 11097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:47:59.914229 11097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:47:59.918218 11097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:47:59.918502 11097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:47:59.919301 11097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:47:59.920456 11097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:47:59.920560 11097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:59.920683 11097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:47:59.920758 11097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:47:59.931476 11097 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:40283
I20260923 14:47:59.933090 11097 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:47:59.935118 11256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:40283 every 8 connection(s)
I20260923 14:47:59.939095 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 11097
I20260923 14:47:59.939601 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:54251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:33945
--local_ip_for_outbound_sockets=127.30.2.194
--tserver_master_addrs=127.30.2.254:42455
--webserver_port=44303
--webserver_interface=127.30.2.194
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.2.212:54251/kms with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:47:59 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:47:59.966749 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:59.937258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:42455 (local address 127.30.2.193:52553)
0923 14:47:59.939056 (+ 1798us) negotiation.cc:107] Waiting for socket to connect
0923 14:47:59.939110 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 14:47:59.940565 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:47:59.944546 (+ 3981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:47:59.944557 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:47:59.944911 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:47:59.945577 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:59.945599 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:59.947970 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:59.947974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:59.949455 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:47:59.949467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:47:59.955470 (+ 6003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:47:59.955475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:47:59.955724 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:59.956553 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:47:59.956577 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:47:59.959695 (+ 3118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:47:59.962733 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:59.962740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:59.962758 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:59.963118 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:59.963585 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:47:59.963589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:47:59.963593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:47:59.963822 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:47:59.964715 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:47:59.964722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:47:59.965036 (+ 314us) client_negotiation.cc:769] Sending connection context
0923 14:47:59.965217 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 14:47:59.965635 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1583,"thread_start_us":153,"threads_started":1}
I20260923 14:47:59.969174 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:47:59.937190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:52553 (local address 127.30.2.254:42455)
0923 14:47:59.937697 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0923 14:47:59.937719 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 14:47:59.939675 (+ 1956us) server_negotiation.cc:407] Connection header received
0923 14:47:59.941335 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:47:59.941351 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:47:59.941797 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:47:59.943982 (+ 2185us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:47:59.945794 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:59.947761 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:59.949631 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:47:59.955232 (+ 5601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:47:59.959995 (+ 4763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:47:59.960235 (+ 240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:47:59.960242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:47:59.960263 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:47:59.960295 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:47:59.962589 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:59.963273 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:59.963282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:59.963290 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:59.963451 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:47:59.963958 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:47:59.963962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:47:59.963964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:47:59.964581 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:47:59.967768 (+ 3187us) server_negotiation.cc:1035] Waiting for connection context
0923 14:47:59.967962 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 14:47:59.968266 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":118,"threads_started":1}
I20260923 14:47:59.972275 11257 heartbeater.cc:347] Connected to a master server at 127.30.2.254:42455
I20260923 14:47:59.972553 11257 heartbeater.cc:464] Registering TS with master...
I20260923 14:47:59.973147 11257 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
W20260923 14:47:59.975567 11257 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:42455 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 14:47:59.975695 11257 heartbeater.cc:464] Registering TS with master...
I20260923 14:47:59.975901 11257 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
I20260923 14:47:59.976827 11257 heartbeater.cc:464] Registering TS with master...
I20260923 14:47:59.977011 11257 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
W20260923 14:47:59.977785 11257 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:48:00.089679 11261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:00.089972 11261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:00.090046 11261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:00.094404 11261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:00.094499 11261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:00.094606 11261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:48:00.100751 11261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.2.212:54251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:33945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=44303
--tserver_master_addrs=127.30.2.254:42455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:00.102273 11261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:00.103746 11261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:00.115849 11269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:00.116763 11266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:00.117475 11261 server_base.cc:1034] running on GCE node
W20260923 14:48:00.119726 11267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:00.120270 11261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:00.121025 11261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:00.122196 11261 hybrid_clock.cc:648] HybridClock initialized: now 1790174880122160 us; error 53 us; skew 500 ppm
Sep 23 14:48:00 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:00.125052 11261 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:48:00.126041 11261 webserver.cc:460] Webserver started at http://127.30.2.194:44303/ using document root <none> and password file <none>
I20260923 14:48:00.126780 11261 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:00.126932 11261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 14:48:00 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:48:00.156100 11261 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:48:00.158799 11277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:00.160346 11261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:48:00.160538 11261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "0ece16c017e44d20bea64604b0e37ff8"
format_stamp: "Formatted at 2026-09-23 14:47:58 on dist-test-slave-wjf2"
server_key: "4ad666868e6d188177d289f408017a6d"
server_key_iv: "f811d82728f22554eb7f3f6d610c1d49"
server_key_version: "kuduclusterkey@0"
I20260923 14:48:00.161352 11261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:00.203891 11261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:00.207140 11261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:00.207345 11261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:00.208040 11261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:00.209169 11261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:00.209278 11261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:00.209375 11261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:00.209439 11261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:00.220821 11261 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:33945
I20260923 14:48:00.220845 11391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:33945 every 8 connection(s)
I20260923 14:48:00.221972 11261 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:48:00.227604 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 11261
I20260923 14:48:00.228145 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.2.212:54251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:36215
--local_ip_for_outbound_sockets=127.30.2.195
--tserver_master_addrs=127.30.2.254:42455
--webserver_port=39689
--webserver_interface=127.30.2.195
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.2.212:54251/kms with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:48:00 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:00.244240 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:00.223933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:50971 (local address 127.30.2.254:42455)
0923 14:48:00.224096 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:00.224100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:00.225018 (+ 918us) server_negotiation.cc:407] Connection header received
0923 14:48:00.226425 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:00.226438 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:00.226500 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:00.226556 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:00.230929 (+ 4373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:00.231760 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:00.233521 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:00.233878 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:00.233954 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:00.239966 (+ 6012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:00.239971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:00.239973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:00.240016 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:00.241780 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:00.242402 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:00.242405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:00.242407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:00.242492 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:00.242875 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:00.242879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:00.242880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:00.243121 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:00.243201 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:00.243904 (+ 703us) server_negotiation.cc:299] Negotiation successful
0923 14:48:00.244066 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 14:48:00.246194 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:00.224409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:42455 (local address 127.30.2.194:50971)
0923 14:48:00.224869 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:00.224921 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:00.226168 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:00.227013 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:00.227022 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:00.230027 (+ 3005us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:00.230737 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:00.230759 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:00.231980 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:00.231983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:00.233379 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:00.233390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:00.234179 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:00.234182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:00.234375 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:00.237942 (+ 3567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:00.237964 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:00.239709 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:00.241915 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:00.241922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:00.241941 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:00.242285 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:00.242600 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:00.242603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:00.242605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:00.242774 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:00.243268 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:00.243274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:00.243601 (+ 327us) client_negotiation.cc:769] Sending connection context
0923 14:48:00.244709 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0923 14:48:00.245088 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":139,"threads_started":1}
I20260923 14:48:00.247578 11392 heartbeater.cc:347] Connected to a master server at 127.30.2.254:42455
I20260923 14:48:00.247900 11392 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:00.248543 11392 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
W20260923 14:48:00.250350 11392 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:42455 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 14:48:00.250486 11392 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:00.250646 11392 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
I20260923 14:48:00.251607 11392 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:00.251768 11392 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
W20260923 14:48:00.252640 11392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:48:00.378310 11395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:00.378572 11395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:00.378670 11395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:00.383082 11395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:00.383164 11395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:00.383296 11395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:48:00.389668 11395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.2.212:54251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:36215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=39689
--tserver_master_addrs=127.30.2.254:42455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:00.390962 11395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:00.392274 11395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:00.404230 11403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:00.404230 11400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:00.404814 11395 server_base.cc:1034] running on GCE node
W20260923 14:48:00.407104 11401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:00.407588 11395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:00.408435 11395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:00.409617 11395 hybrid_clock.cc:648] HybridClock initialized: now 1790174880409583 us; error 45 us; skew 500 ppm
Sep 23 14:48:00 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:00.412964 11395 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:48:00.414045 11395 webserver.cc:460] Webserver started at http://127.30.2.195:39689/ using document root <none> and password file <none>
I20260923 14:48:00.414675 11395 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:00.414789 11395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 14:48:00 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:48:00.433113 11395 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 14:48:00.440892 11411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:00.442653 11395 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260923 14:48:00.442785 11395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "a19b5e42b81d42c6819768cf43bb7ff0"
format_stamp: "Formatted at 2026-09-23 14:47:59 on dist-test-slave-wjf2"
server_key: "cbcd4fde02e93c135856ec992db48949"
server_key_iv: "68b7c35b17d3f0b0af064b9d59344245"
server_key_version: "kuduclusterkey@0"
I20260923 14:48:00.443498 11395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:00.462318 11395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:00.465939 11395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:00.466138 11395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:00.466928 11395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:00.468060 11395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:00.468127 11395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:00.468182 11395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:00.468216 11395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:00.479753 11395 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:36215
I20260923 14:48:00.480275 11524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:36215 every 8 connection(s)
I20260923 14:48:00.481127 11395 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:48:00.481630 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 11395
Sep 23 14:48:00 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:00.494599 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:00.482805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.30.2.254:42455)
0923 14:48:00.482980 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:00.482990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:00.485292 (+ 2302us) server_negotiation.cc:407] Connection header received
0923 14:48:00.485341 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:00.485344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:00.485390 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:00.485482 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:00.486544 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:00.487352 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:00.488694 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:00.489004 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:00.489105 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:00.490849 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:00.490853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:00.490856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:00.490881 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:00.492692 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:00.493160 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:00.493167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:00.493168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:00.493263 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:00.493573 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:00.493577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:00.493586 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:00.493809 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:00.494015 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:00.494192 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 14:48:00.494316 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"spinlock_wait_cycles":3328}
I20260923 14:48:00.498035 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:00.483834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:39231 (local address 127.30.2.254:42455)
0923 14:48:00.484097 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:00.484102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:00.485025 (+ 923us) server_negotiation.cc:407] Connection header received
0923 14:48:00.486291 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:00.486294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:00.486341 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:00.486399 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:00.488003 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:00.488749 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:00.490435 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:00.490789 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:00.490859 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:00.493729 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:00.493734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:00.493736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:00.493763 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:00.495665 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:00.496307 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:00.496310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:00.496312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:00.496398 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:00.496799 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:00.496803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:00.496805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:00.497020 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:00.497122 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:00.497758 (+ 636us) server_negotiation.cc:299] Negotiation successful
0923 14:48:00.497872 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
I20260923 14:48:00.499111 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:00.484386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:42455 (local address 127.30.2.195:39231)
0923 14:48:00.484872 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:00.484924 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:00.486052 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:00.486802 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:00.486811 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:00.487137 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:00.487809 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:00.487831 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:00.488888 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:00.488891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:00.490294 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:00.490306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:00.491082 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:00.491088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:00.491303 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:00.491879 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:00.491901 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:00.493560 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:00.495838 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:00.495844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:00.495859 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:00.496183 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:00.496509 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:00.496513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:00.496515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:00.496686 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:00.497151 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:00.497158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:00.497454 (+ 296us) client_negotiation.cc:769] Sending connection context
0923 14:48:00.497669 (+ 215us) client_negotiation.cc:240] Negotiation successful
0923 14:48:00.497974 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":152,"threads_started":1}
I20260923 14:48:00.500294 11525 heartbeater.cc:347] Connected to a master server at 127.30.2.254:42455
I20260923 14:48:00.500563 11525 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:00.501121 11525 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
I20260923 14:48:00.502681 11059 ts_manager.cc:194] Registered new tserver with Master: a19b5e42b81d42c6819768cf43bb7ff0 (127.30.2.195:36215)
I20260923 14:48:00.504305 11059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:39231
Sep 23 14:48:00 dist-test-slave-wjf2 krb5kdc[10062](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790174880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:00.978039 11257 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:00.978441 11257 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
I20260923 14:48:00.979344 11059 ts_manager.cc:194] Registered new tserver with Master: 37ca786467604d079a1855bd1a5b1f02 (127.30.2.193:40283)
I20260923 14:48:00.980239 11059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:52553
Sep 23 14:48:01 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
I20260923 14:48:01.252883 11392 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:01.253240 11392 heartbeater.cc:511] Master 127.30.2.254:42455 requested a full tablet report, sending...
I20260923 14:48:01.254113 11059 ts_manager.cc:194] Registered new tserver with Master: 0ece16c017e44d20bea64604b0e37ff8 (127.30.2.194:33945)
I20260923 14:48:01.254958 11059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:50971
I20260923 14:48:01.261535 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:48:01.273675 11547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.263289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.30.2.254:42455)
0923 14:48:01.263544 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:01.263548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:01.264150 (+ 602us) server_negotiation.cc:407] Connection header received
0923 14:48:01.264348 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:01.264352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:01.264402 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:01.264459 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:01.265481 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.266344 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.267608 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.267991 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.268072 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.269911 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:01.269915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:01.269918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:01.269946 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:01.271696 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:01.272300 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:01.272304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:01.272306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:01.272404 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:01.272716 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:01.272719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:01.272721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:01.272935 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:01.273071 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:01.273348 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 14:48:01.273484 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
I20260923 14:48:01.277948 11059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:48:01.282747 11059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:48:01.302551 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.294862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:36215 (local address 127.0.0.1:45276)
0923 14:48:01.296453 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:01.296511 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:01.296631 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:01.297860 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:01.297866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:01.297890 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:01.298024 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:01.298031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.299477 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.299480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:01.300842 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:01.300854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.302028 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.302032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:01.302160 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.302202 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 14:48:01.302286 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 14:48:01.302394 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1422,"thread_start_us":115,"threads_started":1}
I20260923 14:48:01.303483 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.295692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.30.2.195:36215)
0923 14:48:01.297389 (+ 1697us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:01.297396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:01.297419 (+ 23us) server_negotiation.cc:407] Connection header received
0923 14:48:01.297493 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:01.297510 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:01.297697 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:01.298171 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:01.298297 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.299347 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.301113 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.301892 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.303004 (+ 1112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.303155 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:01.303259 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 14:48:01.303349 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1597,"thread_start_us":99,"threads_started":1}
I20260923 14:48:01.304329 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.295929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:40283 (local address 127.0.0.1:42864)
0923 14:48:01.297091 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:01.297102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:01.297163 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:01.297968 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:01.297972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:01.297991 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:01.298096 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:01.298103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.300371 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.300375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:01.301682 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:01.301695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.303978 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.303981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:01.304067 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.304090 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 14:48:01.304151 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 14:48:01.304211 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"thread_start_us":93,"threads_started":1}
I20260923 14:48:01.307301 11455 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba183c0272042318ff6deaed52fd045 (DEFAULT_TABLE table=test-table [id=576d01433b5a47bab41dafab83e5f4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:01.308204 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.296157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.30.2.193:40283)
0923 14:48:01.297274 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:01.297283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:01.297310 (+ 27us) server_negotiation.cc:407] Connection header received
0923 14:48:01.297397 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:01.297403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:01.297609 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:01.297756 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:01.298235 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.300232 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.301840 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.303640 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.303748 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.307881 (+ 4133us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:01.307946 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 14:48:01.308048 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":64,"threads_started":1}
I20260923 14:48:01.308815 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba183c0272042318ff6deaed52fd045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:01.310456 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.294922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:33945 (local address 127.0.0.1:47146)
0923 14:48:01.296871 (+ 1949us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:01.296882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:01.296948 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:01.305427 (+ 8479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:01.305431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:01.305452 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:01.305554 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:01.305564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.307760 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.307767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:01.308893 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:01.308906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.310030 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.310034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:01.310130 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.310160 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 14:48:01.310218 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 14:48:01.310290 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1891,"thread_start_us":88,"threads_started":1}
I20260923 14:48:01.311233 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.295102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.30.2.194:33945)
0923 14:48:01.304898 (+ 9796us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:01.304905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:01.304946 (+ 41us) server_negotiation.cc:407] Connection header received
0923 14:48:01.305032 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:01.305038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:01.305255 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:01.305761 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:01.305931 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.307627 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.309070 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.309868 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.310773 (+ 905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.310931 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:01.310988 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 14:48:01.311083 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9672,"thread_start_us":95,"threads_started":1}
I20260923 14:48:01.311260 11191 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba183c0272042318ff6deaed52fd045 (DEFAULT_TABLE table=test-table [id=576d01433b5a47bab41dafab83e5f4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:01.312362 11191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba183c0272042318ff6deaed52fd045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:01.314049 11318 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba183c0272042318ff6deaed52fd045 (DEFAULT_TABLE table=test-table [id=576d01433b5a47bab41dafab83e5f4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:01.315032 11318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba183c0272042318ff6deaed52fd045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:01.318382 11558 tablet_bootstrap.cc:492] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: Bootstrap starting.
I20260923 14:48:01.320807 11558 tablet_bootstrap.cc:654] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:01.321681 11558 log.cc:824] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:01.323416 11560 tablet_bootstrap.cc:492] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02: Bootstrap starting.
I20260923 14:48:01.323844 11558 tablet_bootstrap.cc:492] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: No bootstrap required, opened a new log
I20260923 14:48:01.324036 11558 ts_tablet_manager.cc:1397] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:48:01.325696 11560 tablet_bootstrap.cc:654] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:01.326581 11560 log.cc:824] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:01.327353 11561 tablet_bootstrap.cc:492] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: Bootstrap starting.
I20260923 14:48:01.327447 11558 raft_consensus.cc:348] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 36215 } } peers { permanent_uuid: "0ece16c017e44d20bea64604b0e37ff8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 33945 } } peers { permanent_uuid: "37ca786467604d079a1855bd1a5b1f02" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40283 } }
I20260923 14:48:01.327716 11558 raft_consensus.cc:374] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:01.327798 11558 raft_consensus.cc:729] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19b5e42b81d42c6819768cf43bb7ff0, State: Initialized, Role: FOLLOWER
I20260923 14:48:01.328287 11558 consensus_queue.cc:260] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 36215 } } peers { permanent_uuid: "0ece16c017e44d20bea64604b0e37ff8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 33945 } } peers { permanent_uuid: "37ca786467604d079a1855bd1a5b1f02" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40283 } }
I20260923 14:48:01.329139 11558 ts_tablet_manager.cc:1428] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260923 14:48:01.329588 11525 heartbeater.cc:503] Master 127.30.2.254:42455 was elected leader, sending a full tablet report...
I20260923 14:48:01.329888 11561 tablet_bootstrap.cc:654] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:01.330835 11561 log.cc:824] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:01.331513 11560 tablet_bootstrap.cc:492] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02: No bootstrap required, opened a new log
I20260923 14:48:01.331750 11560 ts_tablet_manager.cc:1397] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 14:48:01.333940 11561 tablet_bootstrap.cc:492] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: No bootstrap required, opened a new log
I20260923 14:48:01.334172 11561 ts_tablet_manager.cc:1397] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 14:48:01.335748 11560 raft_consensus.cc:348] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 36215 } } peers { permanent_uuid: "0ece16c017e44d20bea64604b0e37ff8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 33945 } } peers { permanent_uuid: "37ca786467604d079a1855bd1a5b1f02" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40283 } }
I20260923 14:48:01.336091 11560 raft_consensus.cc:374] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:01.336198 11560 raft_consensus.cc:729] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ca786467604d079a1855bd1a5b1f02, State: Initialized, Role: FOLLOWER
I20260923 14:48:01.336736 11560 consensus_queue.cc:260] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 36215 } } peers { permanent_uuid: "0ece16c017e44d20bea64604b0e37ff8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 33945 } } peers { permanent_uuid: "37ca786467604d079a1855bd1a5b1f02" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40283 } }
I20260923 14:48:01.337800 11560 ts_tablet_manager.cc:1428] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:48:01.338169 11257 heartbeater.cc:503] Master 127.30.2.254:42455 was elected leader, sending a full tablet report...
I20260923 14:48:01.338431 11561 raft_consensus.cc:348] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 36215 } } peers { permanent_uuid: "0ece16c017e44d20bea64604b0e37ff8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 33945 } } peers { permanent_uuid: "37ca786467604d079a1855bd1a5b1f02" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40283 } }
I20260923 14:48:01.338721 11561 raft_consensus.cc:374] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:01.338769 11561 raft_consensus.cc:729] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece16c017e44d20bea64604b0e37ff8, State: Initialized, Role: FOLLOWER
I20260923 14:48:01.340031 11561 consensus_queue.cc:260] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 36215 } } peers { permanent_uuid: "0ece16c017e44d20bea64604b0e37ff8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 33945 } } peers { permanent_uuid: "37ca786467604d079a1855bd1a5b1f02" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40283 } }
I20260923 14:48:01.340946 11392 heartbeater.cc:503] Master 127.30.2.254:42455 was elected leader, sending a full tablet report...
I20260923 14:48:01.341185 11561 ts_tablet_manager.cc:1428] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260923 14:48:01.415858 11563 raft_consensus.cc:482] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:48:01.416124 11563 raft_consensus.cc:504] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 36215 } } peers { permanent_uuid: "0ece16c017e44d20bea64604b0e37ff8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 33945 } } peers { permanent_uuid: "37ca786467604d079a1855bd1a5b1f02" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40283 } }
I20260923 14:48:01.417635 11563 leader_election.cc:288] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ece16c017e44d20bea64604b0e37ff8 (127.30.2.194:33945), 37ca786467604d079a1855bd1a5b1f02 (127.30.2.193:40283)
I20260923 14:48:01.423266 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.418319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:35473 (local address 127.30.2.194:33945)
0923 14:48:01.418480 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:01.418489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:01.418625 (+ 136us) server_negotiation.cc:407] Connection header received
0923 14:48:01.418846 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:01.418850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:01.418900 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:01.418953 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:01.419407 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.420273 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.421909 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.422533 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.422631 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.422683 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:01.423056 (+ 373us) server_negotiation.cc:299] Negotiation successful
0923 14:48:01.423135 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:48:01.423713 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.417898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:33945 (local address 127.30.2.195:35473)
0923 14:48:01.418525 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:01.418540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:01.418716 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:01.419114 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:01.419117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:01.419135 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:01.419276 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:01.419282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.420422 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.420425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:01.421759 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:01.421773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.422798 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.422802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:01.422903 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.422926 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 14:48:01.423462 (+ 536us) client_negotiation.cc:240] Negotiation successful
0923 14:48:01.423549 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":133,"threads_started":1}
I20260923 14:48:01.424320 11338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba183c0272042318ff6deaed52fd045" candidate_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ece16c017e44d20bea64604b0e37ff8" is_pre_election: true
I20260923 14:48:01.424635 11338 raft_consensus.cc:2454] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a19b5e42b81d42c6819768cf43bb7ff0 in term 0.
I20260923 14:48:01.424798 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.418462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:60787 (local address 127.30.2.193:40283)
0923 14:48:01.418590 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:01.418594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:01.419906 (+ 1312us) server_negotiation.cc:407] Connection header received
0923 14:48:01.420143 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:01.420147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:01.420210 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:01.420267 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:01.420757 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.421520 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.423008 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.423591 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.423694 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.423751 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:01.424524 (+ 773us) server_negotiation.cc:299] Negotiation successful
0923 14:48:01.424632 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:48:01.425168 11414 leader_election.cc:302] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ece16c017e44d20bea64604b0e37ff8, a19b5e42b81d42c6819768cf43bb7ff0; no voters:
I20260923 14:48:01.425463 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.417992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:40283 (local address 127.30.2.195:60787)
0923 14:48:01.419807 (+ 1815us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:01.419822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:01.419996 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:01.420429 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:01.420432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:01.420452 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:01.420600 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:01.420607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.421682 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.421685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:01.422828 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:01.422839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.424237 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.424241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:01.424349 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.424377 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 14:48:01.425208 (+ 831us) client_negotiation.cc:240] Negotiation successful
0923 14:48:01.425286 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1730,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260923 14:48:01.425469 11563 raft_consensus.cc:2781] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:48:01.425806 11563 raft_consensus.cc:482] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:48:01.425866 11563 raft_consensus.cc:3037] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:01.426306 11211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba183c0272042318ff6deaed52fd045" candidate_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ca786467604d079a1855bd1a5b1f02" is_pre_election: true
I20260923 14:48:01.426580 11211 raft_consensus.cc:2454] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a19b5e42b81d42c6819768cf43bb7ff0 in term 0.
I20260923 14:48:01.427207 11563 raft_consensus.cc:504] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 36215 } } peers { permanent_uuid: "0ece16c017e44d20bea64604b0e37ff8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 33945 } } peers { permanent_uuid: "37ca786467604d079a1855bd1a5b1f02" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40283 } }
I20260923 14:48:01.427580 11563 leader_election.cc:288] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [CANDIDATE]: Term 1 election: Requested vote from peers 0ece16c017e44d20bea64604b0e37ff8 (127.30.2.194:33945), 37ca786467604d079a1855bd1a5b1f02 (127.30.2.193:40283)
I20260923 14:48:01.428061 11338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba183c0272042318ff6deaed52fd045" candidate_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ece16c017e44d20bea64604b0e37ff8"
I20260923 14:48:01.428061 11211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba183c0272042318ff6deaed52fd045" candidate_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ca786467604d079a1855bd1a5b1f02"
I20260923 14:48:01.428177 11211 raft_consensus.cc:3037] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:01.428187 11338 raft_consensus.cc:3037] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:01.429421 11338 raft_consensus.cc:2454] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a19b5e42b81d42c6819768cf43bb7ff0 in term 1.
I20260923 14:48:01.429570 11211 raft_consensus.cc:2454] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a19b5e42b81d42c6819768cf43bb7ff0 in term 1.
I20260923 14:48:01.429718 11414 leader_election.cc:302] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ece16c017e44d20bea64604b0e37ff8, a19b5e42b81d42c6819768cf43bb7ff0; no voters:
I20260923 14:48:01.429876 11563 raft_consensus.cc:2781] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:48:01.430068 11563 raft_consensus.cc:686] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 1 LEADER]: Becoming Leader. State: Replica: a19b5e42b81d42c6819768cf43bb7ff0, State: Running, Role: LEADER
I20260923 14:48:01.430444 11563 consensus_queue.cc:237] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 36215 } } peers { permanent_uuid: "0ece16c017e44d20bea64604b0e37ff8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 33945 } } peers { permanent_uuid: "37ca786467604d079a1855bd1a5b1f02" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40283 } }
I20260923 14:48:01.434248 11058 catalog_manager.cc:5495] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a19b5e42b81d42c6819768cf43bb7ff0 (127.30.2.195). New cstate: current_term: 1 leader_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19b5e42b81d42c6819768cf43bb7ff0" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 36215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ece16c017e44d20bea64604b0e37ff8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 33945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37ca786467604d079a1855bd1a5b1f02" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40283 } health_report { overall_health: UNKNOWN } } }
W20260923 14:48:01.437381 11258 tablet.cc:2369] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:48:01.460462 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.454963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.30.2.195:36215)
0923 14:48:01.455084 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:01.455105 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:01.455330 (+ 225us) server_negotiation.cc:407] Connection header received
0923 14:48:01.455434 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:01.455438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:01.455515 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:01.455632 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:48:01.456127 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.456939 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.458277 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:01.458635 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:01.458750 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:01.459192 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:48:01.459746 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:48:01.459943 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:01.460257 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 14:48:01.460317 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:48:01.469328 11211 raft_consensus.cc:1261] T 8ba183c0272042318ff6deaed52fd045 P 37ca786467604d079a1855bd1a5b1f02 [term 1 FOLLOWER]: Refusing update from remote peer a19b5e42b81d42c6819768cf43bb7ff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:01.469794 11338 raft_consensus.cc:1261] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8 [term 1 FOLLOWER]: Refusing update from remote peer a19b5e42b81d42c6819768cf43bb7ff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:01.470227 11563 consensus_queue.cc:1035] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37ca786467604d079a1855bd1a5b1f02" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:48:01.470628 11569 consensus_queue.cc:1035] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ece16c017e44d20bea64604b0e37ff8" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 33945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 14:48:01.473961 11393 tablet.cc:2369] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 14:48:01 dist-test-slave-wjf2 krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for HTTP/127.30.2.212@KRBTEST.COM
W20260923 14:48:01.483170 11526 tablet.cc:2369] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:48:01.519644 11576 mvcc.cc:205] Tried to move back new op lower bound from 7332556314487652352 to 7332556314342539264. Current Snapshot: MvccSnapshot[applied={T|T < 7332556314487652352}]
I20260923 14:48:01.520002 11574 mvcc.cc:205] Tried to move back new op lower bound from 7332556314487652352 to 7332556314342539264. Current Snapshot: MvccSnapshot[applied={T|T < 7332556314487652352}]
I20260923 14:48:01.527846 11059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:48:01.528066 11059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:48:01.531363 11059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8788a18b41914418adecf5cf0b16cc87: Sending DeleteTablet for 3 replicas of tablet 8ba183c0272042318ff6deaed52fd045
I20260923 14:48:01.533056 11191 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ba183c0272042318ff6deaed52fd045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:01 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42864
I20260923 14:48:01.533555 11455 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ba183c0272042318ff6deaed52fd045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:01 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:45276
I20260923 14:48:01.534682 11318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ba183c0272042318ff6deaed52fd045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:01 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:47146
I20260923 14:48:01.535851 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 11097
I20260923 14:48:01.540709 11585 tablet_replica.cc:330] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: stopping tablet replica
I20260923 14:48:01.541180 11585 raft_consensus.cc:2229] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:48:01.541622 11585 raft_consensus.cc:2258] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:01.543504 11585 ts_tablet_manager.cc:1905] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:01.543975 11584 tablet_replica.cc:330] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: stopping tablet replica
I20260923 14:48:01.544274 11584 raft_consensus.cc:2229] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:48:01.544730 11584 raft_consensus.cc:2258] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:01.547135 11585 ts_tablet_manager.cc:1918] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:01.547278 11585 log.cc:1196] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/8ba183c0272042318ff6deaed52fd045
I20260923 14:48:01.547681 11585 ts_tablet_manager.cc:1939] T 8ba183c0272042318ff6deaed52fd045 P 0ece16c017e44d20bea64604b0e37ff8: Deleting consensus metadata
I20260923 14:48:01.549320 11045 catalog_manager.cc:4841] TS 0ece16c017e44d20bea64604b0e37ff8 (127.30.2.194:33945): tablet 8ba183c0272042318ff6deaed52fd045 (table test-table [id=576d01433b5a47bab41dafab83e5f4b5]) successfully deleted
W20260923 14:48:01.550716 11043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:40283 (error 108)
I20260923 14:48:01.551894 11584 ts_tablet_manager.cc:1905] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:01.555064 11584 ts_tablet_manager.cc:1918] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:01.555182 11584 log.cc:1196] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/8ba183c0272042318ff6deaed52fd045
I20260923 14:48:01.555563 11584 ts_tablet_manager.cc:1939] T 8ba183c0272042318ff6deaed52fd045 P a19b5e42b81d42c6819768cf43bb7ff0: Deleting consensus metadata
I20260923 14:48:01.556926 11046 catalog_manager.cc:4841] TS a19b5e42b81d42c6819768cf43bb7ff0 (127.30.2.195:36215): tablet 8ba183c0272042318ff6deaed52fd045 (table test-table [id=576d01433b5a47bab41dafab83e5f4b5]) successfully deleted
I20260923 14:48:01.559451 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:01.559064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:40283 (local address 127.0.0.1:42870)
0923 14:48:01.559291 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:01.559359 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:40283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260923 14:48:01.559655 11043 catalog_manager.cc:4584] TS 37ca786467604d079a1855bd1a5b1f02 (127.30.2.193:40283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ba183c0272042318ff6deaed52fd045: Network error: Client connection negotiation failed: client connection to 127.30.2.193:40283: connect: Connection refused (error 111)
I20260923 14:48:01.560491 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 11261
I20260923 14:48:01.570521 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 11395
I20260923 14:48:01.579067 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 11026
I20260923 14:48:01.587175 30731 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 14:48:01.932670 30731 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 14:48:01.932785 30731 mini_ranger.cc:66] Stopping Ranger...
I20260923 14:48:02.276969 30731 mini_ranger.cc:68] Stopped Ranger
2026-09-23 14:48:02.277 UTC [10092] LOG: received smart shutdown request
2026-09-23 14:48:02.281 UTC [10092] LOG: background worker "logical replication launcher" (PID 10101) exited with exit code 1
2026-09-23 14:48:02.281 UTC [10096] LOG: shutting down
2026-09-23 14:48:02.465 UTC [10092] LOG: database system is shut down
2026-09-23T14:48:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48739 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:48:02 dist-test-slave-wjf2 krb5kdc[11596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:48:02 dist-test-slave-wjf2 krb5kdc[11596](info): set up 2 sockets
Sep 23 14:48:02 dist-test-slave-wjf2 krb5kdc[11596](info): commencing operation
krb5kdc: starting...
W20260923 14:48:03.629663 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:48:03 dist-test-slave-wjf2 krb5kdc[11596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:48:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:48:03.754978 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:43663
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:43663,127.30.2.253:45673,127.30.2.252:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:03.867767 11609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:03.868243 11609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:03.868335 11609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:03.872442 11609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:48:03.872524 11609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:03.872562 11609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:03.872622 11609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:48:03.872663 11609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:48:03.878207 11609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:43663,127.30.2.253:45673,127.30.2.252:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:43663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:03.879477 11609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:03.880723 11609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:03.890596 11614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:03.890587 11617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:03.890980 11609 server_base.cc:1034] running on GCE node
W20260923 14:48:03.890587 11615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:03.891666 11609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:03.893015 11609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:03.894188 11609 hybrid_clock.cc:648] HybridClock initialized: now 1790174883894157 us; error 44 us; skew 500 ppm
Sep 23 14:48:03 dist-test-slave-wjf2 krb5kdc[11596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174883, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:03.897271 11609 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:48:03.898240 11609 webserver.cc:460] Webserver started at http://127.30.2.254:35385/ using document root <none> and password file <none>
I20260923 14:48:03.898864 11609 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:03.898957 11609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:03.899216 11609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:03.901757 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "0a94c27fdab84a42b98cd86c7495cb93"
format_stamp: "Formatted at 2026-09-23 14:48:03 on dist-test-slave-wjf2"
server_key: "506a4b2283b93b1afbdee5fd3a68d405"
server_key_iv: "404cf18c8230c26fb0869fa37190f834"
server_key_version: "encryptionkey@0"
I20260923 14:48:03.902401 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "0a94c27fdab84a42b98cd86c7495cb93"
format_stamp: "Formatted at 2026-09-23 14:48:03 on dist-test-slave-wjf2"
server_key: "506a4b2283b93b1afbdee5fd3a68d405"
server_key_iv: "404cf18c8230c26fb0869fa37190f834"
server_key_version: "encryptionkey@0"
I20260923 14:48:03.906587 11609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260923 14:48:03.909322 11624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:03.910693 11609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 14:48:03.910845 11609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "0a94c27fdab84a42b98cd86c7495cb93"
format_stamp: "Formatted at 2026-09-23 14:48:03 on dist-test-slave-wjf2"
server_key: "506a4b2283b93b1afbdee5fd3a68d405"
server_key_iv: "404cf18c8230c26fb0869fa37190f834"
server_key_version: "encryptionkey@0"
I20260923 14:48:03.910972 11609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:03.929183 11609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:03.933019 11609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:03.933264 11609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:03.941327 11609 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:43663
I20260923 14:48:03.941365 11676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:43663 every 8 connection(s)
I20260923 14:48:03.942378 11609 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:48:03.945526 11677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:03.949644 11677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:48:03.951673 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 11609
I20260923 14:48:03.951826 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:48:03.952132 30731 external_mini_cluster.cc:1428] Setting key 7a406108a9931130d1f4cfd71042fe2f
WARNING: no policy specified for kudu/127.30.2.253@KRBTEST.COM; defaulting to no policy
Sep 23 14:48:03 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174883, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.253@KRBTEST.COM" created.
I20260923 14:48:03.966236 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:03.952298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:43663 (local address 127.0.0.1:33020)
0923 14:48:03.952921 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:03.952962 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:03.954002 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:03.954817 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:03.954826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:03.955313 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:03.955585 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:03.955599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:03.956854 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:03.956856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:03.957610 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:03.957620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:03.958305 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:03.958309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:03.958525 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:03.959169 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:03.959188 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:03.960769 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:03.962804 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:03.962812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:03.962822 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:03.963237 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:03.963623 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:03.963627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:03.963629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:03.963823 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:03.964496 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:03.964507 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:03.964707 (+ 200us) client_negotiation.cc:769] Sending connection context
0923 14:48:03.964908 (+ 201us) client_negotiation.cc:240] Negotiation successful
0923 14:48:03.965312 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":20,"spinlock_wait_cycles":29952,"thread_start_us":193,"threads_started":1}
I20260923 14:48:03.966241 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:03.952297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.30.2.254:43663)
0923 14:48:03.952766 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:03.952787 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:03.953076 (+ 289us) server_negotiation.cc:407] Connection header received
0923 14:48:03.954370 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:03.954375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:03.954585 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:03.954746 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:03.955748 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:03.956706 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:03.957737 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:03.958121 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:03.958408 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:03.960973 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:03.960980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:03.960993 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:03.961025 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:03.962683 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:03.963368 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:03.963373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:03.963375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:03.963490 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:03.963927 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:03.963930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:03.963931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:03.964367 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:03.964485 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:03.965096 (+ 611us) server_negotiation.cc:299] Negotiation successful
0923 14:48:03.965415 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":313,"thread_start_us":155,"threads_started":1}
I20260923 14:48:03.969309 11677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:48:03.970163 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:03.969878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:45673 (local address 127.0.0.1:33878)
0923 14:48:03.970012 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:03.970095 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 14:48:03.970399 11626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
I20260923 14:48:03.971294 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:03.971052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:45673 (local address 127.0.0.1:33888)
0923 14:48:03.971172 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:03.971214 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 14:48:03.971558 11677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.253:45673: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab.
Entry for principal kudu/127.30.2.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab.
I20260923 14:48:03.983448 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.253:45673
--webserver_interface=127.30.2.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab
--principal=kudu/127.30.2.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:43663,127.30.2.253:45673,127.30.2.252:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:48:04.020902 11677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } attempt: 1
I20260923 14:48:04.021845 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.021464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:45673 (local address 127.0.0.1:33890)
0923 14:48:04.021673 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.021750 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 14:48:04.022636 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.022394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:45673 (local address 127.0.0.1:33896)
0923 14:48:04.022505 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.022566 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 14:48:04.022782 11677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.253:45673: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
I20260923 14:48:04.091055 11677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } attempt: 2
I20260923 14:48:04.091951 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.091564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:45673 (local address 127.0.0.1:33912)
0923 14:48:04.091733 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.091864 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 14:48:04.092757 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.092541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:45673 (local address 127.0.0.1:33916)
0923 14:48:04.092639 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.092702 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 14:48:04.092957 11677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.253:45673: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
W20260923 14:48:04.096176 11684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:04.096431 11684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:04.096524 11684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:04.100514 11684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:48:04.100584 11684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:04.100633 11684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:04.100673 11684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:48:04.100708 11684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:48:04.105997 11684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:43663,127.30.2.253:45673,127.30.2.252:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab
--principal=kudu/127.30.2.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.253:45673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.2.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:04.107095 11684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:04.108203 11684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:04.116554 11690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:04.116626 11684 server_base.cc:1034] running on GCE node
W20260923 14:48:04.116570 11692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:04.116551 11689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:04.117270 11684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:04.118357 11684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:04.119550 11684 hybrid_clock.cc:648] HybridClock initialized: now 1790174884119535 us; error 25 us; skew 500 ppm
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:04.122264 11684 init.cc:377] Logged in from keytab as kudu/127.30.2.253@KRBTEST.COM (short username kudu)
I20260923 14:48:04.123149 11684 webserver.cc:460] Webserver started at http://127.30.2.253:42809/ using document root <none> and password file <none>
I20260923 14:48:04.123690 11684 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:04.123780 11684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:04.124042 11684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:04.125982 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/data/instance:
uuid: "0f757b8d111c410ca283b38702aa43ed"
format_stamp: "Formatted at 2026-09-23 14:48:04 on dist-test-slave-wjf2"
server_key: "1118fa60dc884beedef719993a6c98c0"
server_key_iv: "b490e3f0b65b0f3a991a67b21384500c"
server_key_version: "encryptionkey@0"
I20260923 14:48:04.126560 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/wal/instance:
uuid: "0f757b8d111c410ca283b38702aa43ed"
format_stamp: "Formatted at 2026-09-23 14:48:04 on dist-test-slave-wjf2"
server_key: "1118fa60dc884beedef719993a6c98c0"
server_key_iv: "b490e3f0b65b0f3a991a67b21384500c"
server_key_version: "encryptionkey@0"
I20260923 14:48:04.130191 11684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:48:04.132503 11699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:04.133699 11684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:48:04.133837 11684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/wal
uuid: "0f757b8d111c410ca283b38702aa43ed"
format_stamp: "Formatted at 2026-09-23 14:48:04 on dist-test-slave-wjf2"
server_key: "1118fa60dc884beedef719993a6c98c0"
server_key_iv: "b490e3f0b65b0f3a991a67b21384500c"
server_key_version: "encryptionkey@0"
I20260923 14:48:04.133944 11684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:04.182627 11684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:04.184116 11677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } attempt: 3
I20260923 14:48:04.185207 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.184911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:45673 (local address 127.0.0.1:33920)
0923 14:48:04.185059 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.185124 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 14:48:04.185802 11684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:04.185868 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.185641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:45673 (local address 127.0.0.1:33932)
0923 14:48:04.185748 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.185811 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 14:48:04.186019 11677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.253:45673: Network error: Client connection negotiation failed: client connection to 127.30.2.253:45673: connect: Connection refused (error 111)
I20260923 14:48:04.186029 11684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:04.193380 11684 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.253:45673
I20260923 14:48:04.193416 11751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.253:45673 every 8 connection(s)
I20260923 14:48:04.194605 11684 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
I20260923 14:48:04.198007 11752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:04.200019 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 11684
I20260923 14:48:04.200132 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-1/wal/instance
I20260923 14:48:04.200419 30731 external_mini_cluster.cc:1428] Setting key 3b32d04af6a261c4f4dd33b31046b2ea
I20260923 14:48:04.201786 11752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.2.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.252@KRBTEST.COM" created.
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:04.218381 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.204107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.30.2.254:43663)
0923 14:48:04.204268 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.204274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.205437 (+ 1163us) server_negotiation.cc:407] Connection header received
0923 14:48:04.206854 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.206859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.206920 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.207019 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.208604 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.209301 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.210841 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.211166 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.211279 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.213956 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.213960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.213962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.213989 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.215691 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.216222 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.216228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.216230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.216318 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.216674 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.216677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.216679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.216864 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.216969 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.218113 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.218240 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:48:04.218806 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.204532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:43663 (local address 127.0.0.1:33022)
0923 14:48:04.205203 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.205298 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:04.206555 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:04.207219 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:04.207228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:04.207720 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:04.208441 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.208466 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.209424 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.209427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.210684 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.210694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.211318 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.211321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.211519 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.212191 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:04.212208 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:04.213791 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:04.215835 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.215841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.215855 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.216116 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.216421 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.216424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.216426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.216571 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.216968 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:04.216975 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:04.217229 (+ 254us) client_negotiation.cc:769] Sending connection context
0923 14:48:04.217408 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 14:48:04.217777 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":131,"threads_started":1}
I20260923 14:48:04.220356 11752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab.
Entry for principal kudu/127.30.2.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab.
I20260923 14:48:04.227277 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.252:40229
--webserver_interface=127.30.2.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab
--principal=kudu/127.30.2.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:43663,127.30.2.253:45673,127.30.2.252:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:48:04.230180 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.220863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:45673 (local address 127.0.0.1:33936)
0923 14:48:04.221424 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.221436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:04.221560 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:04.222000 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:04.222003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:04.222190 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:04.222298 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.222305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.223372 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.223378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.223976 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.223985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.224628 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.224632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.224753 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.225250 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:04.225263 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:04.226578 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:04.228610 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.228616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.228618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.228802 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.229094 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.229099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.229101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.229189 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.229639 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:04.229642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:04.229725 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 14:48:04.229852 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 14:48:04.229976 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":25}
I20260923 14:48:04.230198 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.220975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.30.2.253:45673)
0923 14:48:04.221315 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.221330 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.221561 (+ 231us) server_negotiation.cc:407] Connection header received
0923 14:48:04.221659 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.221664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.221890 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.222024 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.222400 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.223258 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.224085 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.224509 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.224664 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.226729 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.226738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.226749 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.226770 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.228496 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.228922 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.228933 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.228934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.229002 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.229282 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.229285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.229287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.229531 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.229686 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.229932 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.230056 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
I20260923 14:48:04.232292 11752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:48:04.233114 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.232782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:53986)
0923 14:48:04.232926 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.233021 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 14:48:04.233328 11703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
I20260923 14:48:04.234215 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.233945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:53990)
0923 14:48:04.234077 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.234137 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 14:48:04.234414 11752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:40229: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
I20260923 14:48:04.283717 11752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } attempt: 1
I20260923 14:48:04.284654 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.284309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:54002)
0923 14:48:04.284488 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.284559 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 14:48:04.285404 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.285172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:54004)
0923 14:48:04.285293 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.285344 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 14:48:04.285549 11752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:40229: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
I20260923 14:48:04.329190 11677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } attempt: 4
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174883, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
I20260923 14:48:04.338513 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.329820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.30.2.253:45673)
0923 14:48:04.329950 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.329955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.330046 (+ 91us) server_negotiation.cc:407] Connection header received
0923 14:48:04.330217 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.330220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.330275 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.330361 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.330822 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.331692 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.332526 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.332782 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.332894 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.335153 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.335162 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.335164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.335187 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.336739 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.337229 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.337234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.337236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.337312 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.337605 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.337608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.337611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.337854 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.337958 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.338264 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.338375 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:48:04.338517 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.329721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:45673 (local address 127.0.0.1:33952)
0923 14:48:04.329929 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.329946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:04.330100 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:04.330388 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:04.330391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:04.330585 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:04.330699 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.330705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.331841 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.331845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.332395 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.332403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.332897 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.332900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.333010 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.333571 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:04.333588 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:04.335018 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:04.336841 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.336845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.336848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.337134 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.337401 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.337405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.337407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.337500 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.337950 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:04.337953 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:04.338093 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 14:48:04.338198 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 14:48:04.338336 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
W20260923 14:48:04.339171 11758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:04.339504 11758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260923 14:48:04.339461 11677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
W20260923 14:48:04.339582 11758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 14:48:04.340152 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.339935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:54018)
0923 14:48:04.340047 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.340095 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 14:48:04.340770 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.340568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:54026)
0923 14:48:04.340666 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.340710 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 14:48:04.340912 11677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:40229: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
W20260923 14:48:04.344002 11758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:48:04.344070 11758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:04.344107 11758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:04.344143 11758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:48:04.344175 11758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:48:04.349280 11758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:43663,127.30.2.253:45673,127.30.2.252:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab
--principal=kudu/127.30.2.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.252:40229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.2.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:04.350349 11758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:04.351353 11758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:48:04.353718 11752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } attempt: 2
I20260923 14:48:04.354419 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.354130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:54038)
0923 14:48:04.354269 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.354331 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 14:48:04.355118 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.354890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:54040)
0923 14:48:04.355007 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.355057 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 14:48:04.355273 11752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:40229: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
W20260923 14:48:04.360025 11767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:04.360119 11758 server_base.cc:1034] running on GCE node
W20260923 14:48:04.360028 11765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:04.360042 11764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:04.360872 11758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:04.361908 11758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:04.363094 11758 hybrid_clock.cc:648] HybridClock initialized: now 1790174884363077 us; error 24 us; skew 500 ppm
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:04.365681 11758 init.cc:377] Logged in from keytab as kudu/127.30.2.252@KRBTEST.COM (short username kudu)
I20260923 14:48:04.366518 11758 webserver.cc:460] Webserver started at http://127.30.2.252:42019/ using document root <none> and password file <none>
I20260923 14:48:04.367084 11758 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:04.367172 11758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:04.367394 11758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:04.369402 11758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/data/instance:
uuid: "b3b3b5a9ddaa432d9c3a513557ff180d"
format_stamp: "Formatted at 2026-09-23 14:48:04 on dist-test-slave-wjf2"
server_key: "3af27a3dd58daadb440e646fb0d4cb30"
server_key_iv: "1489a8fb3ce300a84afe8af5108ffac8"
server_key_version: "encryptionkey@0"
I20260923 14:48:04.369939 11758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/wal/instance:
uuid: "b3b3b5a9ddaa432d9c3a513557ff180d"
format_stamp: "Formatted at 2026-09-23 14:48:04 on dist-test-slave-wjf2"
server_key: "3af27a3dd58daadb440e646fb0d4cb30"
server_key_iv: "1489a8fb3ce300a84afe8af5108ffac8"
server_key_version: "encryptionkey@0"
I20260923 14:48:04.373489 11758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260923 14:48:04.375707 11774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:04.376878 11758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:48:04.377020 11758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/wal
uuid: "b3b3b5a9ddaa432d9c3a513557ff180d"
format_stamp: "Formatted at 2026-09-23 14:48:04 on dist-test-slave-wjf2"
server_key: "3af27a3dd58daadb440e646fb0d4cb30"
server_key_iv: "1489a8fb3ce300a84afe8af5108ffac8"
server_key_version: "encryptionkey@0"
I20260923 14:48:04.377125 11758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:04.400064 11677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } attempt: 1
I20260923 14:48:04.400856 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.400578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:54050)
0923 14:48:04.400722 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.400778 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 14:48:04.401635 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.401399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:54054)
0923 14:48:04.401534 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.401578 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 14:48:04.401768 11677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:40229: Network error: Client connection negotiation failed: client connection to 127.30.2.252:40229: connect: Connection refused (error 111)
I20260923 14:48:04.402264 11758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:04.405704 11758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:04.405853 11758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:04.412540 11758 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.252:40229
I20260923 14:48:04.412627 11826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.252:40229 every 8 connection(s)
I20260923 14:48:04.413496 11758 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
I20260923 14:48:04.413790 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 11758
I20260923 14:48:04.413923 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174723338690-30731-0/minicluster-data/master-2/wal/instance
I20260923 14:48:04.414320 30731 external_mini_cluster.cc:1428] Setting key 10d85017ffa780f16e244e459afee11a
I20260923 14:48:04.417042 11827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:04.421578 11827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:04.429471 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.415676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.30.2.252:40229)
0923 14:48:04.416131 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.416150 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.416192 (+ 42us) server_negotiation.cc:407] Connection header received
0923 14:48:04.417023 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.417048 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.417484 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.417854 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.418509 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.420003 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.420888 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.421556 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.421721 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.424539 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.424549 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.424562 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.424593 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.426454 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.426871 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.426879 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.426887 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.427033 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.427299 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.427302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.427303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.427971 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.428156 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.428405 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.428705 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":309,"thread_start_us":120,"threads_started":1}
I20260923 14:48:04.433173 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.423497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.30.2.254:43663)
0923 14:48:04.423659 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.423664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.423967 (+ 303us) server_negotiation.cc:407] Connection header received
0923 14:48:04.424198 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.424203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.424257 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.424346 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.425229 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.425849 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.426957 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.427274 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.427367 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.429594 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.429597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.429602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.429628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.431106 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.431849 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.431853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.431857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.431950 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.432257 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.432261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.432262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.432448 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.432560 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.432902 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.433015 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:48:04.433207 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.423440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:43663 (local address 127.0.0.1:33032)
0923 14:48:04.423813 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.423859 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:04.424058 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:04.424395 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:04.424400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:04.424883 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:04.425092 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.425102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.425974 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.425977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.426845 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.426855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.427389 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.427392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.427544 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.428045 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:04.428060 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:04.429404 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:04.431412 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.431420 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.431424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.431698 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.432068 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.432071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.432073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.432150 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.432572 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:04.432580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:04.432702 (+ 122us) client_negotiation.cc:769] Sending connection context
0923 14:48:04.432833 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 14:48:04.433007 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":34,"thread_start_us":131,"threads_started":1}
I20260923 14:48:04.434168 11827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
I20260923 14:48:04.442138 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.431825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.30.2.254:43663)
0923 14:48:04.432147 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.432160 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.432173 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:48:04.432204 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.432207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.432241 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.432355 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.433016 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.433838 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.434820 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.435140 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.435267 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.437977 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.437980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.437983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.438018 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.440224 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.440709 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.440714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.440716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.440817 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.441125 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.441131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.441135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.441384 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.441598 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.441729 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.441859 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":124,"threads_started":1}
I20260923 14:48:04.443743 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.434625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:45673 (local address 127.0.0.1:33966)
0923 14:48:04.434730 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.434744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:04.434815 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:04.435243 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:04.435246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:04.435406 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:04.435486 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.435497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.436531 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.436534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.437197 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.437207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.437967 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.437974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.438080 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.438593 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:04.438610 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:04.439856 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:04.442032 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.442035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.442038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.442302 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.442661 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.442663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.442664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.442780 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.443231 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:04.443235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:04.443334 (+ 99us) client_negotiation.cc:769] Sending connection context
0923 14:48:04.443469 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 14:48:04.443611 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 14:48:04.443838 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.434912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.30.2.253:45673)
0923 14:48:04.435054 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.435058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.435069 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:48:04.435103 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.435105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.435141 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.435230 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.435609 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.436391 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.437328 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.437676 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.437795 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.440000 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.440007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.440010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.440042 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.441897 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.442472 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.442476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.442477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.442572 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.442917 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.442929 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.442931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.443121 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.443240 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.443533 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.443648 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:48:04.444710 11827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:48:04.446429 11752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } attempt: 3
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
I20260923 14:48:04.454582 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.443318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.30.2.253:45673)
0923 14:48:04.443645 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.443654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.443665 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:48:04.443699 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.443701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.443738 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.444124 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.444931 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.445807 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.447083 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.447490 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.447596 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.450426 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.450430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.450435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.450466 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.452416 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.452905 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.452909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.452911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.453015 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.453580 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.453586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.453589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.453766 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.453959 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.454262 (+ 303us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.454377 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
I20260923 14:48:04.454862 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.445224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.30.2.252:40229)
0923 14:48:04.445338 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.445345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.445363 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:48:04.445449 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.445451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.445484 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.445568 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.446035 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.446913 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.448439 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.448884 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.449003 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.451135 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.451139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.451141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.451162 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.452735 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.453177 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.453182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.453184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.453264 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.453548 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.453554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.453556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.453749 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.453851 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.454561 (+ 710us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.454705 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:48:04.455644 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.445178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:54064)
0923 14:48:04.445273 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.445285 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:04.445355 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:04.445604 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:04.445607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:04.445775 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:04.445865 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.445874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.447355 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.447359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.448294 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.448308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.449021 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.449028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.449132 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.449623 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:04.449638 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:04.451009 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:04.452846 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.452850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.452851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.453083 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.453360 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.453364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.453373 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.453452 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.453869 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:04.453875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:04.454335 (+ 460us) client_negotiation.cc:769] Sending connection context
0923 14:48:04.455274 (+ 939us) client_negotiation.cc:240] Negotiation successful
0923 14:48:04.455451 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 14:48:04.457304 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.446844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.30.2.252:40229)
0923 14:48:04.447125 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.447132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.447201 (+ 69us) server_negotiation.cc:407] Connection header received
0923 14:48:04.447314 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.447316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.447345 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.447443 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.447886 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.448989 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.450166 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.450529 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.450654 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.453008 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.453012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.453015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.453043 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.455460 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.455983 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.455987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.456000 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.456095 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.456409 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.456412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.456414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.456655 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.456750 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.457101 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.457188 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":230,"thread_start_us":92,"threads_started":1}
I20260923 14:48:04.457314 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.446824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:54066)
0923 14:48:04.447146 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.447160 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:04.447238 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:04.447460 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:04.447463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:04.447639 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:04.447725 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.447733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.449161 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.449168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.450023 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.450039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.450660 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.450664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.450767 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.451376 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:04.451387 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:04.452891 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:04.455571 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.455574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.455576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.455872 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.456210 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.456213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.456215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.456302 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.456773 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:04.456777 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:04.456878 (+ 101us) client_negotiation.cc:769] Sending connection context
0923 14:48:04.456996 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 14:48:04.457133 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260923 14:48:04.459240 11827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d: Bootstrap starting.
I20260923 14:48:04.460013 11752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed: Bootstrap starting.
I20260923 14:48:04.461975 11827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:04.462189 11752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:04.462813 11827 log.cc:824] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:04.462847 11752 log.cc:824] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:04.464354 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:48:04.465039 11752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed: No bootstrap required, opened a new log
I20260923 14:48:04.465379 11827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d: No bootstrap required, opened a new log
I20260923 14:48:04.469439 11752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } }
I20260923 14:48:04.469712 11752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:04.469697 11677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } attempt: 2
I20260923 14:48:04.469763 11752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f757b8d111c410ca283b38702aa43ed, State: Initialized, Role: FOLLOWER
I20260923 14:48:04.469982 11827 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } }
I20260923 14:48:04.470242 11827 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:04.470310 11827 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b3b5a9ddaa432d9c3a513557ff180d, State: Initialized, Role: FOLLOWER
I20260923 14:48:04.470264 11752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } }
I20260923 14:48:04.470803 11827 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } }
I20260923 14:48:04.471704 11752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:48:04.472671 11827 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [sys.catalog]: configured and running, proceeding with master startup.
Sep 23 14:48:04 dist-test-slave-wjf2 krb5kdc[11596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174883, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
I20260923 14:48:04.476091 11684 master_runner.cc:186] Error getting master registration for 127.30.2.254:43663: OK, instance_id { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" instance_seqno: 1790174883939009 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:48:04.480515 11684 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 14:48:04.482156 11867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:48:04.482234 11867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260923 14:48:04.482550 11859 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:48:04.482607 11859 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:48:04.482910 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.467143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.30.2.252:40229)
0923 14:48:04.467257 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.467261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.468766 (+ 1505us) server_negotiation.cc:407] Connection header received
0923 14:48:04.468933 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.468936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.468978 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.469042 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.475126 (+ 6084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.475834 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.477025 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.477358 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.477429 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.479104 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.479108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.479111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.479138 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.480976 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.481500 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.481504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.481505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.481588 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.481896 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.481899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.481901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.482123 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.482255 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.482625 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.482742 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:48:04.482851 11844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } } }
I20260923 14:48:04.482991 11844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:48:04.483285 11845 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } } }
I20260923 14:48:04.483408 11845 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:48:04.481282 11758 master_runner.cc:186] Error getting master registration for 127.30.2.254:43663: OK, instance_id { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" instance_seqno: 1790174883939009 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:48:04.485074 11758 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:48:04.486124 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.466730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.30.2.254:43663)
0923 14:48:04.466854 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.466859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.467120 (+ 261us) server_negotiation.cc:407] Connection header received
0923 14:48:04.467315 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.467318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.467373 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.467425 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.468612 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.469561 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.470664 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.471058 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.471201 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.473540 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.473545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.473549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.473585 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.475884 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.479132 (+ 3248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.479137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.479140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.479240 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.484667 (+ 5427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.484671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.484673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.484906 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.485710 (+ 804us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.485804 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.485948 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49,"spinlock_wait_cycles":4352}
I20260923 14:48:04.487758 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.471518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.30.2.252:40229)
0923 14:48:04.471651 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.471656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.471670 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:48:04.471707 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.471710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.471750 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.471871 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.472392 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.473180 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.474308 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.474604 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.474707 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.477967 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.477974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.477981 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.478015 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.480210 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.485380 (+ 5170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.485383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.485385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.485495 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.485962 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.485966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.485968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.486171 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.486283 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.487529 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.487654 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:48:04.487967 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.470843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:40229 (local address 127.0.0.1:54086)
0923 14:48:04.471276 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:04.471290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:04.471363 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:04.471960 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:04.471963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:04.472164 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:04.472259 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.472266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.473326 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.473331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.474174 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:04.474185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.474752 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.474757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:04.474860 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.476039 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:04.476061 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:04.477791 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:04.484030 (+ 6239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.484034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.484036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.484304 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.485736 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:04.485739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:04.485741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:04.485842 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:04.486814 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:04.486820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:04.486928 (+ 108us) client_negotiation.cc:769] Sending connection context
0923 14:48:04.487048 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 14:48:04.487190 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"spinlock_wait_cycles":1280}
I20260923 14:48:04.491191 11677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93: Bootstrap starting.
I20260923 14:48:04.491747 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:04.468114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.30.2.253:45673)
0923 14:48:04.468231 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:04.468235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:04.482974 (+ 14739us) server_negotiation.cc:407] Connection header received
0923 14:48:04.483182 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:04.483185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:04.483232 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:04.483287 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:04.484110 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.484878 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.485960 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:04.486300 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:04.486378 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:04.488097 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:04.488100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:04.488103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:04.488132 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:04.489833 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.490351 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.490355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.490357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.490448 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:04.490787 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:04.490792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:04.490796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:04.491009 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:04.491136 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:04.491467 (+ 331us) server_negotiation.cc:299] Negotiation successful
0923 14:48:04.491598 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:48:04.494342 11677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:04.495333 11677 log.cc:824] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:04.497819 11677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93: No bootstrap required, opened a new log
I20260923 14:48:04.502518 11677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } }
I20260923 14:48:04.502779 11677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:04.502857 11677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a94c27fdab84a42b98cd86c7495cb93, State: Initialized, Role: FOLLOWER
I20260923 14:48:04.503393 11677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } }
I20260923 14:48:04.503947 11871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } } }
I20260923 14:48:04.504086 11871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:48:04.504935 11677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 14:48:04.510614 11882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:48:04.510702 11882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:48:04.515237 11609 master_runner.cc:263] No leader master found from master 0a94c27fdab84a42b98cd86c7495cb93
I20260923 14:48:04.515303 11609 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:48:04.646958 11844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:48:04.647181 11844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } }
I20260923 14:48:04.648520 11844 leader_election.cc:288] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a94c27fdab84a42b98cd86c7495cb93 (127.30.2.254:43663), b3b3b5a9ddaa432d9c3a513557ff180d (127.30.2.252:40229)
I20260923 14:48:04.649245 11651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f757b8d111c410ca283b38702aa43ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a94c27fdab84a42b98cd86c7495cb93" is_pre_election: true
I20260923 14:48:04.649565 11651 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f757b8d111c410ca283b38702aa43ed in term 0.
I20260923 14:48:04.650218 11700 leader_election.cc:302] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a94c27fdab84a42b98cd86c7495cb93, 0f757b8d111c410ca283b38702aa43ed; no voters:
I20260923 14:48:04.650990 11801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f757b8d111c410ca283b38702aa43ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" is_pre_election: true
I20260923 14:48:04.651261 11801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f757b8d111c410ca283b38702aa43ed in term 0.
I20260923 14:48:04.651948 11844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:48:04.652032 11844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:48:04.652076 11844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:04.653523 11844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } }
I20260923 14:48:04.653875 11844 leader_election.cc:288] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [CANDIDATE]: Term 1 election: Requested vote from peers 0a94c27fdab84a42b98cd86c7495cb93 (127.30.2.254:43663), b3b3b5a9ddaa432d9c3a513557ff180d (127.30.2.252:40229)
I20260923 14:48:04.654349 11801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f757b8d111c410ca283b38702aa43ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d"
I20260923 14:48:04.654480 11801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:04.655742 11801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f757b8d111c410ca283b38702aa43ed in term 1.
I20260923 14:48:04.656250 11651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f757b8d111c410ca283b38702aa43ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a94c27fdab84a42b98cd86c7495cb93"
I20260923 14:48:04.656378 11651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [term 0 FOLLOWER]: Advancing to term 1
W20260923 14:48:04.657168 11780 tablet.cc:2369] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:48:04.657620 11651 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f757b8d111c410ca283b38702aa43ed in term 1.
I20260923 14:48:04.658027 11700 leader_election.cc:302] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a94c27fdab84a42b98cd86c7495cb93, 0f757b8d111c410ca283b38702aa43ed; no voters:
I20260923 14:48:04.658562 11844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:48:04.658795 11844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [term 1 LEADER]: Becoming Leader. State: Replica: 0f757b8d111c410ca283b38702aa43ed, State: Running, Role: LEADER
I20260923 14:48:04.659157 11844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } }
I20260923 14:48:04.661079 11844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f757b8d111c410ca283b38702aa43ed. Latest consensus state: current_term: 1 leader_uuid: "0f757b8d111c410ca283b38702aa43ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } } }
I20260923 14:48:04.663424 11844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:04.664072 11886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:48:04.665042 11886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:48:04.669940 11651 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [term 1 FOLLOWER]: Refusing update from remote peer 0f757b8d111c410ca283b38702aa43ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:04.669974 11801 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [term 1 FOLLOWER]: Refusing update from remote peer 0f757b8d111c410ca283b38702aa43ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:04.670572 11884 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:48:04.671001 11884 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:48:04.674774 11871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f757b8d111c410ca283b38702aa43ed. Latest consensus state: current_term: 1 leader_uuid: "0f757b8d111c410ca283b38702aa43ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } } }
I20260923 14:48:04.674970 11871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:48:04.674968 11845 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f757b8d111c410ca283b38702aa43ed. Latest consensus state: current_term: 1 leader_uuid: "0f757b8d111c410ca283b38702aa43ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } } }
I20260923 14:48:04.675138 11845 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:48:04.679738 11844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f757b8d111c410ca283b38702aa43ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } } }
I20260923 14:48:04.679870 11844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:04.679996 11844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f757b8d111c410ca283b38702aa43ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } } }
I20260923 14:48:04.680090 11844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:04.681375 11886 catalog_manager.cc:1347] Generated new cluster ID: 5c9b9590d03940c7ab56f3eaf61d9e6b
I20260923 14:48:04.681430 11886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:48:04.682345 11845 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f757b8d111c410ca283b38702aa43ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } } }
I20260923 14:48:04.682454 11845 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:48:04.684195 11871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f757b8d111c410ca283b38702aa43ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a94c27fdab84a42b98cd86c7495cb93" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 43663 } } peers { permanent_uuid: "0f757b8d111c410ca283b38702aa43ed" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 45673 } } peers { permanent_uuid: "b3b3b5a9ddaa432d9c3a513557ff180d" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 40229 } } }
I20260923 14:48:04.684288 11871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:48:04.684741 11887 mvcc.cc:205] Tried to move back new op lower bound from 7332556327601889280 to 7332556327567937536. Current Snapshot: MvccSnapshot[applied={T|T < 7332556327601889280 or (T in {7332556327601889280})}]
I20260923 14:48:04.685091 11889 mvcc.cc:205] Tried to move back new op lower bound from 7332556327601889280 to 7332556327567937536. Current Snapshot: MvccSnapshot[applied={T|T < 7332556327601889280}]
I20260923 14:48:04.686014 11888 mvcc.cc:205] Tried to move back new op lower bound from 7332556327601889280 to 7332556327567937536. Current Snapshot: MvccSnapshot[applied={T|T < 7332556327601889280}]
I20260923 14:48:04.692114 11886 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:48:04.693130 11886 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:48:04.702132 11886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f757b8d111c410ca283b38702aa43ed: Generated new TSK 0
I20260923 14:48:04.702869 11886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:48:05.483665 11867 catalog_manager.cc:1259] Loaded cluster ID: 5c9b9590d03940c7ab56f3eaf61d9e6b
I20260923 14:48:05.483777 11867 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d: loading cluster ID for follower catalog manager: success
I20260923 14:48:05.485651 11867 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d: acquiring CA information for follower catalog manager: success
I20260923 14:48:05.486937 11867 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3b3b5a9ddaa432d9c3a513557ff180d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:48:05.512014 11882 catalog_manager.cc:1259] Loaded cluster ID: 5c9b9590d03940c7ab56f3eaf61d9e6b
I20260923 14:48:05.512091 11882 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93: loading cluster ID for follower catalog manager: success
I20260923 14:48:05.513798 11882 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93: acquiring CA information for follower catalog manager: success
I20260923 14:48:05.515154 11882 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0a94c27fdab84a42b98cd86c7495cb93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:48:05.779738 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 11609
I20260923 14:48:05.786944 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 11684
I20260923 14:48:05.793279 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 11758
2026-09-23T14:48:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3210 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:48:05 dist-test-slave-wjf2 krb5kdc[11901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:48:05 dist-test-slave-wjf2 krb5kdc[11901](info): set up 2 sockets
Sep 23 14:48:05 dist-test-slave-wjf2 krb5kdc[11901](info): commencing operation
krb5kdc: starting...
W20260923 14:48:06.825366 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:48:06 dist-test-slave-wjf2 krb5kdc[11901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:48:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:48:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:48:06.952039 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45445
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:45445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:07.059871 11914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:07.060151 11914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:07.060232 11914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:07.064742 11914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:48:07.064816 11914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:07.064862 11914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:07.064934 11914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:48:07.064968 11914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:48:07.070093 11914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:45445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:07.071071 11914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:07.072180 11914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:07.080379 11920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:07.080570 11919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:07.080542 11922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:07.080623 11914 server_base.cc:1034] running on GCE node
I20260923 14:48:07.081342 11914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:07.082386 11914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:07.083603 11914 hybrid_clock.cc:648] HybridClock initialized: now 1790174887083588 us; error 24 us; skew 500 ppm
Sep 23 14:48:07 dist-test-slave-wjf2 krb5kdc[11901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174887, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:07.086450 11914 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:48:07.087344 11914 webserver.cc:460] Webserver started at http://127.30.2.254:42271/ using document root <none> and password file <none>
I20260923 14:48:07.087988 11914 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:07.088079 11914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:07.088284 11914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:07.090265 11914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "1ea9dd6f043e4497afe4bb9a50daea84"
format_stamp: "Formatted at 2026-09-23 14:48:07 on dist-test-slave-wjf2"
server_key: "69f22e41ba35815b5bac670a11d7ee9a"
server_key_iv: "6bc2fa5e06fb023a46b5200fb0d4c307"
server_key_version: "encryptionkey@0"
I20260923 14:48:07.090811 11914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "1ea9dd6f043e4497afe4bb9a50daea84"
format_stamp: "Formatted at 2026-09-23 14:48:07 on dist-test-slave-wjf2"
server_key: "69f22e41ba35815b5bac670a11d7ee9a"
server_key_iv: "6bc2fa5e06fb023a46b5200fb0d4c307"
server_key_version: "encryptionkey@0"
I20260923 14:48:07.094559 11914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260923 14:48:07.097417 11929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:07.098716 11914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 14:48:07.098918 11914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "1ea9dd6f043e4497afe4bb9a50daea84"
format_stamp: "Formatted at 2026-09-23 14:48:07 on dist-test-slave-wjf2"
server_key: "69f22e41ba35815b5bac670a11d7ee9a"
server_key_iv: "6bc2fa5e06fb023a46b5200fb0d4c307"
server_key_version: "encryptionkey@0"
I20260923 14:48:07.099058 11914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:07.124451 11914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:07.127912 11914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:07.128118 11914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:07.135320 11914 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:45445
I20260923 14:48:07.135335 11981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:45445 every 8 connection(s)
I20260923 14:48:07.136315 11914 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:48:07.137637 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 11914
I20260923 14:48:07.137780 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:48:07.138093 30731 external_mini_cluster.cc:1428] Setting key 43d8046b901fab7171864d203bfdc4b0
I20260923 14:48:07.139443 11982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:07.146179 11982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84: Bootstrap starting.
Sep 23 14:48:07 dist-test-slave-wjf2 krb5kdc[11901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:07.148947 11982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:07.149864 11982 log.cc:824] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:07.152326 11982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84: No bootstrap required, opened a new log
I20260923 14:48:07.153798 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:07.139469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.30.2.254:45445)
0923 14:48:07.139941 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:07.139961 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:07.140004 (+ 43us) server_negotiation.cc:407] Connection header received
0923 14:48:07.140821 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:07.140843 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:07.141242 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:07.141602 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:07.142261 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:07.143738 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:07.144724 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:07.145457 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:07.145616 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:07.148340 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:07.148346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:07.148362 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:07.148392 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:07.150468 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:07.151075 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:07.151081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:07.151087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:07.151220 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:07.151551 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:07.151554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:07.151556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:07.152081 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:07.152260 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:07.152586 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 14:48:07.152885 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":116,"threads_started":1}
I20260923 14:48:07.156669 11982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9dd6f043e4497afe4bb9a50daea84" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45445 } }
I20260923 14:48:07.156965 11982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:07.157047 11982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea9dd6f043e4497afe4bb9a50daea84, State: Initialized, Role: FOLLOWER
I20260923 14:48:07.157040 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:48:07.157459 11982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9dd6f043e4497afe4bb9a50daea84" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45445 } }
I20260923 14:48:07.157614 11982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:48:07.157698 11982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:48:07.157807 11982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:07.158845 11982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9dd6f043e4497afe4bb9a50daea84" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45445 } }
I20260923 14:48:07.159183 11982 leader_election.cc:302] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ea9dd6f043e4497afe4bb9a50daea84; no voters:
I20260923 14:48:07.159484 11982 leader_election.cc:288] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:48:07.159683 11987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:48:07.160074 11987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [term 1 LEADER]: Becoming Leader. State: Replica: 1ea9dd6f043e4497afe4bb9a50daea84, State: Running, Role: LEADER
I20260923 14:48:07.160413 11987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9dd6f043e4497afe4bb9a50daea84" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45445 } }
I20260923 14:48:07.160837 11982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:48:07.162081 11989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ea9dd6f043e4497afe4bb9a50daea84. Latest consensus state: current_term: 1 leader_uuid: "1ea9dd6f043e4497afe4bb9a50daea84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9dd6f043e4497afe4bb9a50daea84" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45445 } } }
I20260923 14:48:07.162240 11989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:07.162628 11988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ea9dd6f043e4497afe4bb9a50daea84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9dd6f043e4497afe4bb9a50daea84" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45445 } } }
I20260923 14:48:07.162741 11988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84 [sys.catalog]: This master's current role is: LEADER
W20260923 14:48:07.165601 12002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:48:07.165674 12002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:48:07.165766 11999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:48:07.166594 11999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:48:07.172509 11999 catalog_manager.cc:1347] Generated new cluster ID: aac6508750d44f6f95613f54e7186c13
I20260923 14:48:07.172585 11999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:48:07.183077 11999 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:48:07.184105 11999 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:48:07.193951 11999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ea9dd6f043e4497afe4bb9a50daea84: Generated new TSK 0
I20260923 14:48:07.194931 11999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:48:07.213783 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 11914
2026-09-23T14:48:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1419 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:48:07 dist-test-slave-wjf2 krb5kdc[12009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:48:07 dist-test-slave-wjf2 krb5kdc[12009](info): set up 2 sockets
Sep 23 14:48:07 dist-test-slave-wjf2 krb5kdc[12009](info): commencing operation
krb5kdc: starting...
W20260923 14:48:08.333279 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:48:08 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:48:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:48:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:48:08.454655 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:33037
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:33037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:08.562198 12022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:08.562433 12022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:08.562472 12022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:08.566356 12022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:48:08.566404 12022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:08.566423 12022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:08.566439 12022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:48:08.566454 12022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:48:08.571467 12022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:33037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:08.572476 12022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:08.573424 12022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:08.581837 12027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:08.581818 12028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:08.581847 12030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:08.582175 12022 server_base.cc:1034] running on GCE node
I20260923 14:48:08.582690 12022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:08.583761 12022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:08.585006 12022 hybrid_clock.cc:648] HybridClock initialized: now 1790174888584979 us; error 43 us; skew 500 ppm
Sep 23 14:48:08 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174888, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:08.591426 12022 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:48:08.592355 12022 webserver.cc:460] Webserver started at http://127.30.2.254:44579/ using document root <none> and password file <none>
I20260923 14:48:08.592917 12022 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:08.593036 12022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:08.593254 12022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:08.595260 12022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "f61def0074ec4acfb053f1d3c37444b5"
format_stamp: "Formatted at 2026-09-23 14:48:08 on dist-test-slave-wjf2"
server_key: "9f1cfbb6f9665de4213a83df5061c83f"
server_key_iv: "756f7d756de30f7ba4080a911a4dc901"
server_key_version: "encryptionkey@0"
I20260923 14:48:08.595897 12022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "f61def0074ec4acfb053f1d3c37444b5"
format_stamp: "Formatted at 2026-09-23 14:48:08 on dist-test-slave-wjf2"
server_key: "9f1cfbb6f9665de4213a83df5061c83f"
server_key_iv: "756f7d756de30f7ba4080a911a4dc901"
server_key_version: "encryptionkey@0"
I20260923 14:48:08.599668 12022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:48:08.602142 12037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:08.603379 12022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:48:08.603526 12022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "f61def0074ec4acfb053f1d3c37444b5"
format_stamp: "Formatted at 2026-09-23 14:48:08 on dist-test-slave-wjf2"
server_key: "9f1cfbb6f9665de4213a83df5061c83f"
server_key_iv: "756f7d756de30f7ba4080a911a4dc901"
server_key_version: "encryptionkey@0"
I20260923 14:48:08.603637 12022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:08.619266 12022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:08.622191 12022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:08.622399 12022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:08.629464 12022 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:33037
I20260923 14:48:08.629786 12089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:33037 every 8 connection(s)
I20260923 14:48:08.630466 12022 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:48:08.633240 12090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:08.638729 12090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Bootstrap starting.
I20260923 14:48:08.640045 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 12022
I20260923 14:48:08.640206 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:48:08.640484 30731 external_mini_cluster.cc:1428] Setting key b536d19cd34c77ce0b10a9f57a4be215
I20260923 14:48:08.641517 12090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:08.642251 12090 log.cc:824] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:08.644436 12090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: No bootstrap required, opened a new log
I20260923 14:48:08.648644 12090 raft_consensus.cc:348] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:08.648926 12090 raft_consensus.cc:374] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:08.648994 12090 raft_consensus.cc:729] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61def0074ec4acfb053f1d3c37444b5, State: Initialized, Role: FOLLOWER
I20260923 14:48:08.649446 12090 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:08.649606 12090 raft_consensus.cc:388] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 23 14:48:08 dist-test-slave-wjf2 krb5kdc[12009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:08.649695 12090 raft_consensus.cc:482] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:48:08.649814 12090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:08.650864 12090 raft_consensus.cc:504] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:08.651257 12090 leader_election.cc:302] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f61def0074ec4acfb053f1d3c37444b5; no voters:
I20260923 14:48:08.651623 12090 leader_election.cc:288] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:48:08.651679 12095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:48:08.651880 12095 raft_consensus.cc:686] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 1 LEADER]: Becoming Leader. State: Replica: f61def0074ec4acfb053f1d3c37444b5, State: Running, Role: LEADER
I20260923 14:48:08.652230 12095 consensus_queue.cc:237] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:08.652884 12090 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:48:08.653391 12095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f61def0074ec4acfb053f1d3c37444b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } } }
I20260923 14:48:08.653558 12095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:08.653825 12096 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61def0074ec4acfb053f1d3c37444b5. Latest consensus state: current_term: 1 leader_uuid: "f61def0074ec4acfb053f1d3c37444b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } } }
I20260923 14:48:08.653927 12096 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:08.654011 12101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:48:08.656345 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:08.641601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53912 (local address 127.30.2.254:33037)
0923 14:48:08.642098 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:08.642114 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:08.642154 (+ 40us) server_negotiation.cc:407] Connection header received
0923 14:48:08.642979 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:08.643005 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:08.643367 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:08.643641 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:08.644594 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:08.645578 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:08.646802 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:08.647278 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:08.647470 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:08.650355 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:08.650361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:08.650374 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:08.650400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:08.652473 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:08.652907 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:08.652913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:08.652920 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:08.653064 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:08.653395 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:08.653398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:08.653400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:08.653975 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:08.654881 (+ 906us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:08.655032 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 14:48:08.655335 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":159,"threads_started":1}
I20260923 14:48:08.657343 12101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:48:08.662731 12101 catalog_manager.cc:1347] Generated new cluster ID: f7183d6b4f1444d59688d1f83e8c5096
I20260923 14:48:08.662793 12101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:48:08.677376 12101 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:48:08.678294 12101 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:48:08.686105 12101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Generated new TSK 0
I20260923 14:48:08.686769 12101 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:48:08.719344 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33037
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:08.852933 12116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:08.853181 12116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:08.853224 12116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:08.857401 12116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:08.857458 12116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:08.857545 12116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:48:08.862634 12116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:08.863605 12116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:08.864640 12116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:08.874207 12122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:08.874214 12121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:08.874317 12116 server_base.cc:1034] running on GCE node
W20260923 14:48:08.874523 12124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:08.874969 12116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:08.875605 12116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:08.876765 12116 hybrid_clock.cc:648] HybridClock initialized: now 1790174888876749 us; error 26 us; skew 500 ppm
Sep 23 14:48:08 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174888, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:08.879485 12116 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:48:08.880353 12116 webserver.cc:460] Webserver started at http://127.30.2.193:36201/ using document root <none> and password file <none>
I20260923 14:48:08.880870 12116 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:08.880927 12116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:08.881086 12116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:08.883037 12116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "c7889a3e96284cfba0427850c5e08124"
format_stamp: "Formatted at 2026-09-23 14:48:08 on dist-test-slave-wjf2"
server_key: "f6192e5bbfc90f16e5649a79996bf667"
server_key_iv: "7ffefb2df733a38459c934d5f98a7974"
server_key_version: "encryptionkey@0"
I20260923 14:48:08.883541 12116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "c7889a3e96284cfba0427850c5e08124"
format_stamp: "Formatted at 2026-09-23 14:48:08 on dist-test-slave-wjf2"
server_key: "f6192e5bbfc90f16e5649a79996bf667"
server_key_iv: "7ffefb2df733a38459c934d5f98a7974"
server_key_version: "encryptionkey@0"
I20260923 14:48:08.887146 12116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 14:48:08.889503 12131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:08.890769 12116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:48:08.890901 12116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "c7889a3e96284cfba0427850c5e08124"
format_stamp: "Formatted at 2026-09-23 14:48:08 on dist-test-slave-wjf2"
server_key: "f6192e5bbfc90f16e5649a79996bf667"
server_key_iv: "7ffefb2df733a38459c934d5f98a7974"
server_key_version: "encryptionkey@0"
I20260923 14:48:08.891019 12116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:08.908362 12116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:08.911744 12116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:08.911988 12116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:08.912683 12116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:08.913556 12116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:08.913610 12116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:08.913681 12116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:08.913712 12116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:08.922825 12116 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:42783
I20260923 14:48:08.922948 12244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:42783 every 8 connection(s)
I20260923 14:48:08.923795 12116 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:48:08.926362 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 12116
I20260923 14:48:08.926525 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:48:08.926934 30731 external_mini_cluster.cc:1428] Setting key dc33047195e3253ccf4eb053b341dc4d
Sep 23 14:48:08 dist-test-slave-wjf2 krb5kdc[12009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174888, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:48:08.941516 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:08.925685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:59591 (local address 127.30.2.254:33037)
0923 14:48:08.925870 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:08.925874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:08.926853 (+ 979us) server_negotiation.cc:407] Connection header received
0923 14:48:08.928689 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:08.928692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:08.928734 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:08.928796 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:08.930576 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:08.931425 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:08.933325 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:08.933565 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:08.933648 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:08.937034 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:08.937040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:08.937045 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:08.937079 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:08.939193 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:08.939703 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:08.939706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:08.939707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:08.939844 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:08.940240 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:08.940242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:08.940243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:08.940437 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:08.940522 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:08.941262 (+ 740us) server_negotiation.cc:299] Negotiation successful
0923 14:48:08.941378 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
I20260923 14:48:08.942462 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:08.926067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:59591)
0923 14:48:08.926635 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:08.926722 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:08.928425 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:08.929033 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:08.929042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:08.929621 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:08.930387 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:08.930413 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:08.931572 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:08.931575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:08.933191 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:08.933207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:08.933737 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:08.933745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:08.934062 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:08.934814 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:08.934843 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:08.936786 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:08.939322 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:08.939327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:08.939339 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:08.939591 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:08.939994 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:08.939997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:08.939999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:08.940142 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:08.940573 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:08.940579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:08.940947 (+ 368us) client_negotiation.cc:769] Sending connection context
0923 14:48:08.941137 (+ 190us) client_negotiation.cc:240] Negotiation successful
0923 14:48:08.941407 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":121,"threads_started":1}
I20260923 14:48:08.943962 12245 heartbeater.cc:347] Connected to a master server at 127.30.2.254:33037
I20260923 14:48:08.944248 12245 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:08.944782 12245 heartbeater.cc:511] Master 127.30.2.254:33037 requested a full tablet report, sending...
I20260923 14:48:08.946509 12054 ts_manager.cc:194] Registered new tserver with Master: c7889a3e96284cfba0427850c5e08124 (127.30.2.193:42783)
I20260923 14:48:08.947830 12054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:59591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:48:08.959182 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33037
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:09.071233 12250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:09.071527 12250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:09.071599 12250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:09.075525 12250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:09.075575 12250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:09.075696 12250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:48:09.080797 12250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:09.081730 12250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:09.082700 12250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:09.091864 12255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:09.091909 12258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:09.091988 12250 server_base.cc:1034] running on GCE node
W20260923 14:48:09.091879 12256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:09.092568 12250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:09.093219 12250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:09.094414 12250 hybrid_clock.cc:648] HybridClock initialized: now 1790174889094393 us; error 31 us; skew 500 ppm
Sep 23 14:48:09 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174889, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:09.097252 12250 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:48:09.098258 12250 webserver.cc:460] Webserver started at http://127.30.2.194:46339/ using document root <none> and password file <none>
I20260923 14:48:09.098838 12250 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:09.098950 12250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:09.099221 12250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:09.101297 12250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "5e1fd031be08463cbda4dfc74eaef121"
format_stamp: "Formatted at 2026-09-23 14:48:09 on dist-test-slave-wjf2"
server_key: "15ae710df0904708cfd60deddd3ae378"
server_key_iv: "e35ec314212e874ea89062d17ba477b8"
server_key_version: "encryptionkey@0"
I20260923 14:48:09.101878 12250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "5e1fd031be08463cbda4dfc74eaef121"
format_stamp: "Formatted at 2026-09-23 14:48:09 on dist-test-slave-wjf2"
server_key: "15ae710df0904708cfd60deddd3ae378"
server_key_iv: "e35ec314212e874ea89062d17ba477b8"
server_key_version: "encryptionkey@0"
I20260923 14:48:09.105726 12250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:48:09.108315 12265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:09.109611 12250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:48:09.109771 12250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "5e1fd031be08463cbda4dfc74eaef121"
format_stamp: "Formatted at 2026-09-23 14:48:09 on dist-test-slave-wjf2"
server_key: "15ae710df0904708cfd60deddd3ae378"
server_key_iv: "e35ec314212e874ea89062d17ba477b8"
server_key_version: "encryptionkey@0"
I20260923 14:48:09.109918 12250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:09.150436 12250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:09.153451 12250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:09.153656 12250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:09.154446 12250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:09.155381 12250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:09.155450 12250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:09.155531 12250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:09.155584 12250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:09.164191 12250 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:45189
I20260923 14:48:09.164227 12378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:45189 every 8 connection(s)
I20260923 14:48:09.165035 12250 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:48:09.165486 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 12250
I20260923 14:48:09.165598 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:48:09.165906 30731 external_mini_cluster.cc:1428] Setting key 3f845b27daba6d22e5fc27c7f710c952
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
Sep 23 14:48:09 dist-test-slave-wjf2 krb5kdc[12009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174889, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
I20260923 14:48:09.180454 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.166909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:41393 (local address 127.30.2.254:33037)
0923 14:48:09.167046 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:09.167051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:09.168122 (+ 1071us) server_negotiation.cc:407] Connection header received
0923 14:48:09.169950 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:09.169955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:09.170011 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:09.170115 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:09.171252 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.171844 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.173784 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.174004 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.174094 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.176865 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:09.176871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:09.176878 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:09.176901 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:09.178236 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:09.178721 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:09.178726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:09.178728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:09.178816 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:09.179220 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:09.179223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:09.179224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:09.179392 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:09.179469 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:09.180119 (+ 650us) server_negotiation.cc:299] Negotiation successful
0923 14:48:09.180223 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:48:09.181349 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.167383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:41393)
0923 14:48:09.167944 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:09.168005 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:09.169735 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:09.170293 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:09.170300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:09.170654 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:09.171109 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.171122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.171980 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.171983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.173544 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.173560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.174172 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.174179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.174394 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.175024 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:09.175044 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:09.176703 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:09.178347 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:09.178353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:09.178367 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:09.178617 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:09.178929 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:09.178934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:09.178936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:09.179108 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:09.179506 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:09.179515 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:09.179863 (+ 348us) client_negotiation.cc:769] Sending connection context
0923 14:48:09.180094 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 14:48:09.180451 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":117,"threads_started":1}
I20260923 14:48:09.182665 12379 heartbeater.cc:347] Connected to a master server at 127.30.2.254:33037
I20260923 14:48:09.182904 12379 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:09.183359 12379 heartbeater.cc:511] Master 127.30.2.254:33037 requested a full tablet report, sending...
I20260923 14:48:09.184464 12054 ts_manager.cc:194] Registered new tserver with Master: 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189)
I20260923 14:48:09.185245 12054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:41393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:48:09.195950 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33037
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:09.320125 12384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:09.320358 12384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:09.320395 12384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:09.324280 12384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:09.324329 12384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:09.324383 12384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:48:09.329370 12384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:09.330286 12384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:09.331223 12384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:09.340646 12390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:09.340659 12389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:09.340641 12392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:09.340824 12384 server_base.cc:1034] running on GCE node
I20260923 14:48:09.341336 12384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:09.341972 12384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:09.343168 12384 hybrid_clock.cc:648] HybridClock initialized: now 1790174889343153 us; error 26 us; skew 500 ppm
Sep 23 14:48:09 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174889, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:09.345888 12384 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:48:09.346730 12384 webserver.cc:460] Webserver started at http://127.30.2.195:35313/ using document root <none> and password file <none>
I20260923 14:48:09.347287 12384 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:09.347378 12384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:09.347605 12384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:09.349784 12384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "abefccf81418440b8b4bc6edb53e0c14"
format_stamp: "Formatted at 2026-09-23 14:48:09 on dist-test-slave-wjf2"
server_key: "f805607a0e9c6ad5a7a4c3fc07db675a"
server_key_iv: "5ca9b8787e010195716e1677df883db2"
server_key_version: "encryptionkey@0"
I20260923 14:48:09.350349 12384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "abefccf81418440b8b4bc6edb53e0c14"
format_stamp: "Formatted at 2026-09-23 14:48:09 on dist-test-slave-wjf2"
server_key: "f805607a0e9c6ad5a7a4c3fc07db675a"
server_key_iv: "5ca9b8787e010195716e1677df883db2"
server_key_version: "encryptionkey@0"
I20260923 14:48:09.354382 12384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:48:09.356906 12399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:09.358180 12384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:48:09.358315 12384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "abefccf81418440b8b4bc6edb53e0c14"
format_stamp: "Formatted at 2026-09-23 14:48:09 on dist-test-slave-wjf2"
server_key: "f805607a0e9c6ad5a7a4c3fc07db675a"
server_key_iv: "5ca9b8787e010195716e1677df883db2"
server_key_version: "encryptionkey@0"
I20260923 14:48:09.358428 12384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:09.387298 12384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:09.390252 12384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:09.390450 12384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:09.391016 12384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:09.391927 12384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:09.391991 12384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:09.392059 12384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:09.392109 12384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:09.401167 12384 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:41461
I20260923 14:48:09.401194 12513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:41461 every 8 connection(s)
I20260923 14:48:09.402068 12384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
Sep 23 14:48:09 dist-test-slave-wjf2 krb5kdc[12009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174889, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:09.412083 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 12384
I20260923 14:48:09.412226 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:48:09.412505 30731 external_mini_cluster.cc:1428] Setting key d22f4a5024b640ff8d8ee9d62df14d70
I20260923 14:48:09.416595 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.403973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:39705 (local address 127.30.2.254:33037)
0923 14:48:09.404116 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:09.404120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:09.405018 (+ 898us) server_negotiation.cc:407] Connection header received
0923 14:48:09.406426 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:09.406430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:09.406505 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:09.406611 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:09.407974 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.408595 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.409782 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.410009 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.410120 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.412711 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:09.412715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:09.412718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:09.412748 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:09.414422 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:09.415020 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:09.415026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:09.415032 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:09.415109 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:09.415469 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:09.415477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:09.415479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:09.415654 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:09.415762 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:09.416300 (+ 538us) server_negotiation.cc:299] Negotiation successful
0923 14:48:09.416416 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:48:09.417374 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.404406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:39705)
0923 14:48:09.404868 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:09.404910 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:09.406204 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:09.406807 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:09.406814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:09.407265 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:09.407821 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.407845 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.408721 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.408725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.409655 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.409665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.410164 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.410168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.410350 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.410915 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:09.410932 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:09.412517 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:09.414529 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:09.414535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:09.414550 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:09.414894 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:09.415202 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:09.415206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:09.415208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:09.415356 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:09.415760 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:09.415767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:09.416061 (+ 294us) client_negotiation.cc:769] Sending connection context
0923 14:48:09.416245 (+ 184us) client_negotiation.cc:240] Negotiation successful
0923 14:48:09.416563 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":111,"threads_started":1}
I20260923 14:48:09.418460 12514 heartbeater.cc:347] Connected to a master server at 127.30.2.254:33037
I20260923 14:48:09.418706 12514 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:09.419193 12514 heartbeater.cc:511] Master 127.30.2.254:33037 requested a full tablet report, sending...
I20260923 14:48:09.420233 12054 ts_manager.cc:194] Registered new tserver with Master: abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461)
I20260923 14:48:09.420823 12054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:39705
I20260923 14:48:09.426702 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:48:09.435385 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.428713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.30.2.254:33037)
0923 14:48:09.428853 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:09.428858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:09.428983 (+ 125us) server_negotiation.cc:407] Connection header received
0923 14:48:09.429090 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:09.429092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:09.429130 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:09.429196 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:09.429815 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.430305 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.431101 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.431295 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.431365 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.432650 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:09.432653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:09.432655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:09.432676 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:09.433903 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:09.434296 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:09.434298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:09.434300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:09.434368 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:09.434601 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:09.434603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:09.434604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:09.434787 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:09.434886 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:09.435103 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 14:48:09.435187 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:48:09.437989 12054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:48:09.441020 12054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:48:09.456102 12529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.449453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:45189 (local address 127.0.0.1:33236)
0923 14:48:09.450427 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:09.450443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:09.450618 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:09.451094 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:09.451100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:09.451127 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:09.451250 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.451257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.452599 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.452603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.454230 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.454240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.455418 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.455422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.455596 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.455684 (+ 88us) client_negotiation.cc:769] Sending connection context
0923 14:48:09.455832 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 14:48:09.455960 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":316,"thread_start_us":100,"threads_started":1}
I20260923 14:48:09.456180 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.449842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.30.2.194:45189)
0923 14:48:09.450291 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:09.450301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:09.450526 (+ 225us) server_negotiation.cc:407] Connection header received
0923 14:48:09.450763 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:09.450768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:09.450924 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:09.451403 (+ 479us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:09.451584 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.452463 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.454347 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.455282 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.455391 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.455535 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:09.455932 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 14:48:09.456026 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":145,"threads_started":1}
I20260923 14:48:09.456247 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.449435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.30.2.195:41461)
0923 14:48:09.449917 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:09.449930 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:09.450321 (+ 391us) server_negotiation.cc:407] Connection header received
0923 14:48:09.450620 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:09.450626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:09.450839 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:09.450986 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:09.451371 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.452623 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.454513 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.455282 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.455391 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.455535 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:09.455988 (+ 453us) server_negotiation.cc:299] Negotiation successful
0923 14:48:09.456087 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":143,"threads_started":1}
I20260923 14:48:09.457334 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.449948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:42783 (local address 127.0.0.1:36418)
0923 14:48:09.450934 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:09.450944 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:09.451051 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:09.451511 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:09.451514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:09.451530 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:09.451608 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.451618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.452940 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.452944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.454387 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.454398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.456931 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.456935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.457020 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.457045 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 14:48:09.457138 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 14:48:09.457213 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":64,"thread_start_us":73,"threads_started":1}
I20260923 14:48:09.457404 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.450348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.30.2.193:42783)
0923 14:48:09.450834 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:09.450839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:09.451029 (+ 190us) server_negotiation.cc:407] Connection header received
0923 14:48:09.451193 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:09.451197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:09.451364 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:09.451511 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:09.451736 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.452825 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.455599 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.456785 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.456907 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.457084 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:09.457183 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 14:48:09.457279 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":113,"threads_started":1}
I20260923 14:48:09.459017 12313 tablet_service.cc:1467] Processing CreateTablet for tablet 945e0d8637fd41f7a6a2383bf4dd1476 (DEFAULT_TABLE table=test-table [id=6c7ed69c9f9a4633bfc5387b718ab580]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:09.460031 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.449249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:41461 (local address 127.0.0.1:55548)
0923 14:48:09.450192 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:09.450240 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:09.450430 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:09.451179 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:09.451182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:09.451194 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:09.451255 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.451260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.452805 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.452808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.454387 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.454398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.455418 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.455422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.455601 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.455684 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 14:48:09.455817 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 14:48:09.459903 (+ 4086us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":221,"thread_start_us":72,"threads_started":1}
I20260923 14:48:09.460093 12313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945e0d8637fd41f7a6a2383bf4dd1476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:09.462383 12447 tablet_service.cc:1467] Processing CreateTablet for tablet 945e0d8637fd41f7a6a2383bf4dd1476 (DEFAULT_TABLE table=test-table [id=6c7ed69c9f9a4633bfc5387b718ab580]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:09.462393 12179 tablet_service.cc:1467] Processing CreateTablet for tablet 945e0d8637fd41f7a6a2383bf4dd1476 (DEFAULT_TABLE table=test-table [id=6c7ed69c9f9a4633bfc5387b718ab580]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:09.463351 12447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945e0d8637fd41f7a6a2383bf4dd1476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:09.463446 12179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945e0d8637fd41f7a6a2383bf4dd1476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:09.469233 12532 tablet_bootstrap.cc:492] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: Bootstrap starting.
I20260923 14:48:09.469244 12531 tablet_bootstrap.cc:492] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Bootstrap starting.
I20260923 14:48:09.471637 12531 tablet_bootstrap.cc:654] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:09.472070 12532 tablet_bootstrap.cc:654] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:09.472612 12531 log.cc:824] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:09.473304 12532 log.cc:824] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:09.474797 12531 tablet_bootstrap.cc:492] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: No bootstrap required, opened a new log
I20260923 14:48:09.475001 12531 ts_tablet_manager.cc:1397] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:48:09.475283 12532 tablet_bootstrap.cc:492] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: No bootstrap required, opened a new log
I20260923 14:48:09.475464 12532 ts_tablet_manager.cc:1397] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:48:09.475889 12535 tablet_bootstrap.cc:492] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: Bootstrap starting.
I20260923 14:48:09.478263 12535 tablet_bootstrap.cc:654] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:09.478955 12535 log.cc:824] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:09.479398 12531 raft_consensus.cc:348] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:09.479398 12532 raft_consensus.cc:348] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:09.479666 12532 raft_consensus.cc:374] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:09.479666 12531 raft_consensus.cc:374] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:09.479724 12531 raft_consensus.cc:729] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7889a3e96284cfba0427850c5e08124, State: Initialized, Role: FOLLOWER
I20260923 14:48:09.479724 12532 raft_consensus.cc:729] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abefccf81418440b8b4bc6edb53e0c14, State: Initialized, Role: FOLLOWER
I20260923 14:48:09.480206 12532 consensus_queue.cc:260] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:09.480206 12531 consensus_queue.cc:260] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:09.480815 12535 tablet_bootstrap.cc:492] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: No bootstrap required, opened a new log
I20260923 14:48:09.481002 12535 ts_tablet_manager.cc:1397] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:48:09.481199 12531 ts_tablet_manager.cc:1428] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260923 14:48:09.481200 12532 ts_tablet_manager.cc:1428] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 14:48:09.481570 12514 heartbeater.cc:503] Master 127.30.2.254:33037 was elected leader, sending a full tablet report...
I20260923 14:48:09.481639 12245 heartbeater.cc:503] Master 127.30.2.254:33037 was elected leader, sending a full tablet report...
I20260923 14:48:09.484733 12535 raft_consensus.cc:348] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:09.484997 12535 raft_consensus.cc:374] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:09.485064 12535 raft_consensus.cc:729] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1fd031be08463cbda4dfc74eaef121, State: Initialized, Role: FOLLOWER
I20260923 14:48:09.485502 12535 consensus_queue.cc:260] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:09.486119 12379 heartbeater.cc:503] Master 127.30.2.254:33037 was elected leader, sending a full tablet report...
I20260923 14:48:09.486495 12535 ts_tablet_manager.cc:1428] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260923 14:48:09.653059 12515 tablet.cc:2369] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:48:09.666320 12380 tablet.cc:2369] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:48:09.675153 12246 tablet.cc:2369] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:48:09.737226 12538 raft_consensus.cc:482] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:48:09.737545 12538 raft_consensus.cc:504] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:09.738685 12538 leader_election.cc:288] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461), 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189)
I20260923 14:48:09.742769 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.739162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:43237 (local address 127.30.2.195:41461)
0923 14:48:09.739293 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:09.739299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:09.739580 (+ 281us) server_negotiation.cc:407] Connection header received
0923 14:48:09.739734 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:09.739741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:09.739834 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:09.739923 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:09.740198 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.740687 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.741783 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.742179 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.742306 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.742360 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:09.742576 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 14:48:09.742643 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:48:09.742775 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.739051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:58563 (local address 127.30.2.194:45189)
0923 14:48:09.739208 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:09.739213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:09.739471 (+ 258us) server_negotiation.cc:407] Connection header received
0923 14:48:09.739646 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:09.739650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:09.739702 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:09.739806 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:09.740116 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.740636 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.741782 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.742180 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.742303 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.742361 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:09.742594 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 14:48:09.742658 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:48:09.742794 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.738996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:41461 (local address 127.30.2.193:43237)
0923 14:48:09.739477 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:09.739490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:09.739614 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:09.739946 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:09.739948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:09.739964 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:09.740085 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.740090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.740794 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.740800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.741655 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.741665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.742298 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.742302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.742447 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.742466 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 14:48:09.742558 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 14:48:09.742633 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":96,"thread_start_us":115,"threads_started":1}
I20260923 14:48:09.742805 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.738980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:45189 (local address 127.30.2.193:58563)
0923 14:48:09.739368 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:09.739380 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:09.739527 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:09.739817 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:09.739820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:09.739840 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:09.739997 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.740004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.740747 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.740753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.741655 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:09.741664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.742303 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.742307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:09.742469 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.742488 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 14:48:09.742575 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 14:48:09.742645 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":42,"thread_start_us":126,"threads_started":1}
I20260923 14:48:09.743634 12467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945e0d8637fd41f7a6a2383bf4dd1476" candidate_uuid: "c7889a3e96284cfba0427850c5e08124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abefccf81418440b8b4bc6edb53e0c14" is_pre_election: true
I20260923 14:48:09.743623 12333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945e0d8637fd41f7a6a2383bf4dd1476" candidate_uuid: "c7889a3e96284cfba0427850c5e08124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1fd031be08463cbda4dfc74eaef121" is_pre_election: true
I20260923 14:48:09.743953 12333 raft_consensus.cc:2454] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7889a3e96284cfba0427850c5e08124 in term 0.
I20260923 14:48:09.743963 12467 raft_consensus.cc:2454] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7889a3e96284cfba0427850c5e08124 in term 0.
I20260923 14:48:09.744417 12132 leader_election.cc:302] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5e1fd031be08463cbda4dfc74eaef121, abefccf81418440b8b4bc6edb53e0c14, c7889a3e96284cfba0427850c5e08124; no voters:
I20260923 14:48:09.744716 12538 raft_consensus.cc:2781] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:48:09.744798 12538 raft_consensus.cc:482] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:48:09.744828 12538 raft_consensus.cc:3037] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:09.745903 12538 raft_consensus.cc:504] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:09.746296 12538 leader_election.cc:288] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 1 election: Requested vote from peers abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461), 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189)
I20260923 14:48:09.746657 12467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945e0d8637fd41f7a6a2383bf4dd1476" candidate_uuid: "c7889a3e96284cfba0427850c5e08124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abefccf81418440b8b4bc6edb53e0c14"
I20260923 14:48:09.746759 12333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945e0d8637fd41f7a6a2383bf4dd1476" candidate_uuid: "c7889a3e96284cfba0427850c5e08124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1fd031be08463cbda4dfc74eaef121"
I20260923 14:48:09.746799 12467 raft_consensus.cc:3037] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:09.746883 12333 raft_consensus.cc:3037] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:09.748178 12333 raft_consensus.cc:2454] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7889a3e96284cfba0427850c5e08124 in term 1.
I20260923 14:48:09.748178 12467 raft_consensus.cc:2454] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7889a3e96284cfba0427850c5e08124 in term 1.
I20260923 14:48:09.748468 12132 leader_election.cc:302] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1fd031be08463cbda4dfc74eaef121, c7889a3e96284cfba0427850c5e08124; no voters:
I20260923 14:48:09.748644 12538 raft_consensus.cc:2781] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:48:09.748874 12538 raft_consensus.cc:686] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 1 LEADER]: Becoming Leader. State: Replica: c7889a3e96284cfba0427850c5e08124, State: Running, Role: LEADER
I20260923 14:48:09.749210 12538 consensus_queue.cc:237] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:09.752317 12054 catalog_manager.cc:5495] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7889a3e96284cfba0427850c5e08124 (127.30.2.193). New cstate: current_term: 1 leader_uuid: "c7889a3e96284cfba0427850c5e08124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } health_report { overall_health: HEALTHY } } }
I20260923 14:48:09.816732 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.812879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.30.2.193:42783)
0923 14:48:09.813046 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:09.813051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:09.813067 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:48:09.813199 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:09.813203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:09.813264 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:09.813378 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:48:09.813636 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.814220 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.815132 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:09.815379 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:09.815499 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:09.815868 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:48:09.816303 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:48:09.816444 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:09.816494 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 14:48:09.816553 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 14:48:09.822659 12467 raft_consensus.cc:1261] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14 [term 1 FOLLOWER]: Refusing update from remote peer c7889a3e96284cfba0427850c5e08124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:09.822659 12333 raft_consensus.cc:1261] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121 [term 1 FOLLOWER]: Refusing update from remote peer c7889a3e96284cfba0427850c5e08124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:09.823377 12538 consensus_queue.cc:1035] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:48:09.823638 12542 consensus_queue.cc:1035] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:48:09.830595 12545 mvcc.cc:205] Tried to move back new op lower bound from 7332556348706390016 to 7332556348415377408. Current Snapshot: MvccSnapshot[applied={T|T < 7332556348706390016}]
I20260923 14:48:09.831322 12547 mvcc.cc:205] Tried to move back new op lower bound from 7332556348706390016 to 7332556348415377408. Current Snapshot: MvccSnapshot[applied={T|T < 7332556348706390016}]
I20260923 14:48:09.831962 12546 mvcc.cc:205] Tried to move back new op lower bound from 7332556348706390016 to 7332556348415377408. Current Snapshot: MvccSnapshot[applied={T|T < 7332556348706390016}]
I20260923 14:48:09.840749 12054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:48:09.840893 12054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:48:09.843284 12054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Sending DeleteTablet for 3 replicas of tablet 945e0d8637fd41f7a6a2383bf4dd1476
I20260923 14:48:09.844139 12313 tablet_service.cc:1514] Processing DeleteTablet for tablet 945e0d8637fd41f7a6a2383bf4dd1476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:09 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:33236
I20260923 14:48:09.844129 12447 tablet_service.cc:1514] Processing DeleteTablet for tablet 945e0d8637fd41f7a6a2383bf4dd1476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:09 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:55548
I20260923 14:48:09.844488 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 12116
I20260923 14:48:09.844686 12556 tablet_replica.cc:330] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: stopping tablet replica
I20260923 14:48:09.845155 12556 raft_consensus.cc:2229] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:48:09.845501 12556 raft_consensus.cc:2258] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:09.846184 12555 tablet_replica.cc:330] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: stopping tablet replica
I20260923 14:48:09.846509 12555 raft_consensus.cc:2229] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:48:09.846830 12555 raft_consensus.cc:2258] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:09.847154 12556 ts_tablet_manager.cc:1905] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:09.848172 12555 ts_tablet_manager.cc:1905] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:09.849944 12556 ts_tablet_manager.cc:1918] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:09.850039 12556 log.cc:1196] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/945e0d8637fd41f7a6a2383bf4dd1476
I20260923 14:48:09.850344 12556 ts_tablet_manager.cc:1939] T 945e0d8637fd41f7a6a2383bf4dd1476 P 5e1fd031be08463cbda4dfc74eaef121: Deleting consensus metadata
I20260923 14:48:09.850910 12555 ts_tablet_manager.cc:1918] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:09.851017 12555 log.cc:1196] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/945e0d8637fd41f7a6a2383bf4dd1476
I20260923 14:48:09.851338 12555 ts_tablet_manager.cc:1939] T 945e0d8637fd41f7a6a2383bf4dd1476 P abefccf81418440b8b4bc6edb53e0c14: Deleting consensus metadata
I20260923 14:48:09.851632 12038 catalog_manager.cc:4841] TS 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189): tablet 945e0d8637fd41f7a6a2383bf4dd1476 (table test-table [id=6c7ed69c9f9a4633bfc5387b718ab580]) successfully deleted
W20260923 14:48:09.852516 12038 connection.cc:535] client connection to 127.30.2.193:42783 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 14:48:09.852600 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 12250
I20260923 14:48:09.852674 12040 catalog_manager.cc:4841] TS abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461): tablet 945e0d8637fd41f7a6a2383bf4dd1476 (table test-table [id=6c7ed69c9f9a4633bfc5387b718ab580]) successfully deleted
W20260923 14:48:09.852685 12038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 14:48:09.854496 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:09.854255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:42783 (local address 127.0.0.1:36436)
0923 14:48:09.854361 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:09.854427 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260923 14:48:09.854645 12038 catalog_manager.cc:4584] TS c7889a3e96284cfba0427850c5e08124 (127.30.2.193:42783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 945e0d8637fd41f7a6a2383bf4dd1476: Network error: Client connection negotiation failed: client connection to 127.30.2.193:42783: connect: Connection refused (error 111)
I20260923 14:48:09.858804 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 12384
I20260923 14:48:09.864499 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 12022
I20260923 14:48:09.871038 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:33037
--webserver_interface=127.30.2.254
--webserver_port=44579
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:33037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:09.980724 12558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:09.980990 12558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:09.981069 12558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:09.985080 12558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:48:09.985152 12558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:09.985191 12558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:09.985245 12558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:48:09.985280 12558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:48:09.990439 12558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:33037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=44579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:09.991400 12558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:09.992445 12558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:10.007745 12567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:10.007848 12558 server_base.cc:1034] running on GCE node
W20260923 14:48:10.007725 12564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:10.007717 12565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:10.008574 12558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:10.009713 12558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:10.010915 12558 hybrid_clock.cc:648] HybridClock initialized: now 1790174890010879 us; error 49 us; skew 500 ppm
Sep 23 14:48:10 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174890, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:10.013665 12558 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:48:10.014576 12558 webserver.cc:460] Webserver started at http://127.30.2.254:44579/ using document root <none> and password file <none>
I20260923 14:48:10.015156 12558 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:10.015255 12558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:10.019161 12558 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:48:10.021382 12574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:10.022616 12558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:48:10.022769 12558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "f61def0074ec4acfb053f1d3c37444b5"
format_stamp: "Formatted at 2026-09-23 14:48:08 on dist-test-slave-wjf2"
server_key: "9f1cfbb6f9665de4213a83df5061c83f"
server_key_iv: "756f7d756de30f7ba4080a911a4dc901"
server_key_version: "encryptionkey@0"
I20260923 14:48:10.023360 12558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:10.041203 12558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:10.044121 12558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:10.044317 12558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:10.051504 12558 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:33037
I20260923 14:48:10.051800 12626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:33037 every 8 connection(s)
I20260923 14:48:10.052677 12558 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:48:10.056846 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 12558
I20260923 14:48:10.057222 12627 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:48:10.057299 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:42783
--local_ip_for_outbound_sockets=127.30.2.193
--tserver_master_addrs=127.30.2.254:33037
--webserver_port=36201
--webserver_interface=127.30.2.193
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:48:10.058992 12627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Bootstrap starting.
I20260923 14:48:10.076176 12627 log.cc:824] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:10.084977 12627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:48:10.085340 12627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Bootstrap complete.
I20260923 14:48:10.089442 12627 raft_consensus.cc:348] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:10.089841 12627 raft_consensus.cc:729] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61def0074ec4acfb053f1d3c37444b5, State: Initialized, Role: FOLLOWER
I20260923 14:48:10.090329 12627 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:10.090487 12627 raft_consensus.cc:388] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:48:10.090574 12627 raft_consensus.cc:482] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:48:10.090693 12627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:48:10.092619 12627 raft_consensus.cc:504] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:10.092986 12627 leader_election.cc:302] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f61def0074ec4acfb053f1d3c37444b5; no voters:
I20260923 14:48:10.093318 12627 leader_election.cc:288] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 14:48:10.093436 12631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 14:48:10.093685 12631 raft_consensus.cc:686] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 2 LEADER]: Becoming Leader. State: Replica: f61def0074ec4acfb053f1d3c37444b5, State: Running, Role: LEADER
I20260923 14:48:10.093997 12631 consensus_queue.cc:237] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:10.094538 12627 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:48:10.095324 12632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f61def0074ec4acfb053f1d3c37444b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } } }
I20260923 14:48:10.095451 12632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:10.095755 12633 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61def0074ec4acfb053f1d3c37444b5. Latest consensus state: current_term: 2 leader_uuid: "f61def0074ec4acfb053f1d3c37444b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } } }
I20260923 14:48:10.095860 12633 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:10.096159 12640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:48:10.100893 12640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:48:10.101689 12640 catalog_manager.cc:1259] Loaded cluster ID: f7183d6b4f1444d59688d1f83e8c5096
I20260923 14:48:10.101758 12640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:48:10.103420 12640 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 14:48:10.107935 12640 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 14:48:10.113199 12640 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:48:10.113938 12640 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Loaded TSK: 0
I20260923 14:48:10.115154 12640 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:48:10.173988 12629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:10.174221 12629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:10.174260 12629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:10.178294 12629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:10.178341 12629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:10.178393 12629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:48:10.183465 12629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:42783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=36201
--tserver_master_addrs=127.30.2.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:10.184422 12629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:10.185395 12629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:10.194551 12654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:10.194767 12657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:10.194582 12655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:10.195160 12629 server_base.cc:1034] running on GCE node
I20260923 14:48:10.195566 12629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:10.196269 12629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:10.197450 12629 hybrid_clock.cc:648] HybridClock initialized: now 1790174890197427 us; error 40 us; skew 500 ppm
Sep 23 14:48:10 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174890, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:10.200309 12629 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:48:10.201268 12629 webserver.cc:460] Webserver started at http://127.30.2.193:36201/ using document root <none> and password file <none>
I20260923 14:48:10.201826 12629 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:10.201946 12629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:10.205829 12629 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260923 14:48:10.208029 12664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:10.209299 12629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:48:10.209390 12629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "c7889a3e96284cfba0427850c5e08124"
format_stamp: "Formatted at 2026-09-23 14:48:08 on dist-test-slave-wjf2"
server_key: "f6192e5bbfc90f16e5649a79996bf667"
server_key_iv: "7ffefb2df733a38459c934d5f98a7974"
server_key_version: "encryptionkey@0"
I20260923 14:48:10.209939 12629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:10.223966 12629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:10.227550 12629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:10.227748 12629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:10.228397 12629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:10.229488 12671 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 14:48:10.234860 12629 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 14:48:10.234997 12629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260923 14:48:10.235076 12629 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 14:48:10.236840 12629 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 14:48:10.236907 12629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260923 14:48:10.237167 12671 tablet_bootstrap.cc:492] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Bootstrap starting.
I20260923 14:48:10.246688 12629 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:42783
I20260923 14:48:10.246780 12778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:42783 every 8 connection(s)
I20260923 14:48:10.247589 12629 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:48:10.253882 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 12629
I20260923 14:48:10.254319 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:45189
--local_ip_for_outbound_sockets=127.30.2.194
--tserver_master_addrs=127.30.2.254:33037
--webserver_port=46339
--webserver_interface=127.30.2.194
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:48:10.257496 12671 log.cc:824] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Log is configured to *not* fsync() on all Append() calls
Sep 23 14:48:10 dist-test-slave-wjf2 krb5kdc[12009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174890, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:10.265725 12671 tablet_bootstrap.cc:492] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:48:10.266148 12671 tablet_bootstrap.cc:492] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Bootstrap complete.
I20260923 14:48:10.266714 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.249818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:54479 (local address 127.30.2.254:33037)
0923 14:48:10.250197 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:10.250216 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:10.250543 (+ 327us) server_negotiation.cc:407] Connection header received
0923 14:48:10.251681 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:10.251696 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:10.252238 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:10.252644 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:10.253717 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.255409 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.256722 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.257178 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.257358 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.260592 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:10.260600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:10.260617 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:10.260651 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:10.262804 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:10.263490 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:10.263496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:10.263502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:10.263649 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:10.264092 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:10.264096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:10.264097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:10.264532 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:10.264664 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:10.265384 (+ 720us) server_negotiation.cc:299] Negotiation successful
0923 14:48:10.265693 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":126,"threads_started":1}
I20260923 14:48:10.266723 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.249735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:54479)
0923 14:48:10.250346 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:10.250395 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:10.251255 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:10.252715 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:10.252722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:10.253062 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:10.253521 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.253535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.255590 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.255596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.256588 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.256599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.257323 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.257327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.257530 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.258237 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:10.258263 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:10.260345 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:10.262941 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:10.262949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:10.262967 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:10.263347 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:10.263761 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:10.263767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:10.263770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:10.263981 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:10.264666 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:10.264675 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:10.265004 (+ 329us) client_negotiation.cc:769] Sending connection context
0923 14:48:10.265224 (+ 220us) client_negotiation.cc:240] Negotiation successful
0923 14:48:10.265603 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":149,"threads_started":1}
I20260923 14:48:10.267550 12671 ts_tablet_manager.cc:1397] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.009s
I20260923 14:48:10.269729 12779 heartbeater.cc:347] Connected to a master server at 127.30.2.254:33037
I20260923 14:48:10.269963 12779 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:10.270505 12779 heartbeater.cc:511] Master 127.30.2.254:33037 requested a full tablet report, sending...
I20260923 14:48:10.270184 12671 raft_consensus.cc:348] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:10.270663 12671 raft_consensus.cc:729] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7889a3e96284cfba0427850c5e08124, State: Initialized, Role: FOLLOWER
I20260923 14:48:10.271153 12671 consensus_queue.cc:260] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:10.272161 12671 ts_tablet_manager.cc:1428] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260923 14:48:10.273711 12591 ts_manager.cc:194] Registered new tserver with Master: c7889a3e96284cfba0427850c5e08124 (127.30.2.193:42783)
I20260923 14:48:10.277115 12591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:54479
I20260923 14:48:10.279516 12779 heartbeater.cc:503] Master 127.30.2.254:33037 was elected leader, sending a full tablet report...
Sep 23 14:48:10 dist-test-slave-wjf2 krb5kdc[12009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174890, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:48:10.287374 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.277118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.30.2.193:42783)
0923 14:48:10.277438 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:10.277450 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:10.277472 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:48:10.277538 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:10.277543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:10.277738 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:10.277883 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:10.278859 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.279915 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.281100 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.281423 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.281505 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.283912 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:10.283920 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:10.283930 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:10.283951 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:10.285483 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:10.286001 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:10.286006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:10.286007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:10.286081 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:10.286385 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:10.286388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:10.286389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:10.286633 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:10.286768 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:10.287115 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 14:48:10.287255 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":89,"threads_started":1}
I20260923 14:48:10.287374 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.276411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:42783 (local address 127.0.0.1:36440)
0923 14:48:10.276821 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:10.276871 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:10.277010 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:10.278171 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:10.278177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:10.278599 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:10.278724 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.278732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.280062 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.280066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.280979 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.280990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.281531 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.281535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.281660 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.282193 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:10.282207 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:10.283730 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:10.285593 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:10.285602 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:10.285608 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:10.285880 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:10.286168 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:10.286172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:10.286174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:10.286265 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:10.286769 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:10.286775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:10.286889 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 14:48:10.287034 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 14:48:10.287208 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":102,"threads_started":1}
I20260923 14:48:10.288570 12713 tablet_service.cc:1514] Processing DeleteTablet for tablet 945e0d8637fd41f7a6a2383bf4dd1476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:09 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:36440
I20260923 14:48:10.289444 12788 tablet_replica.cc:330] stopping tablet replica
I20260923 14:48:10.289757 12788 raft_consensus.cc:2229] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:48:10.290007 12788 raft_consensus.cc:2258] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:10.291150 12788 ts_tablet_manager.cc:1905] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:10.295625 12788 ts_tablet_manager.cc:1918] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:10.295720 12788 log.cc:1196] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/945e0d8637fd41f7a6a2383bf4dd1476
I20260923 14:48:10.296020 12788 ts_tablet_manager.cc:1939] T 945e0d8637fd41f7a6a2383bf4dd1476 P c7889a3e96284cfba0427850c5e08124: Deleting consensus metadata
I20260923 14:48:10.297883 12575 catalog_manager.cc:4841] TS c7889a3e96284cfba0427850c5e08124 (127.30.2.193:42783): tablet 945e0d8637fd41f7a6a2383bf4dd1476 (table test-table [id=6c7ed69c9f9a4633bfc5387b718ab580]) successfully deleted
W20260923 14:48:10.372530 12783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:10.372771 12783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:10.372848 12783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:10.376832 12783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:10.376880 12783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:10.376965 12783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:48:10.382117 12783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:45189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=46339
--tserver_master_addrs=127.30.2.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:10.383127 12783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:10.384220 12783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:48:10.393709 12783 server_base.cc:1034] running on GCE node
W20260923 14:48:10.393793 12796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:10.393664 12794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:10.393674 12793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:10.394495 12783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:10.395108 12783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:10.396320 12783 hybrid_clock.cc:648] HybridClock initialized: now 1790174890396298 us; error 31 us; skew 500 ppm
Sep 23 14:48:10 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174890, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:10.398874 12783 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:48:10.399624 12783 webserver.cc:460] Webserver started at http://127.30.2.194:46339/ using document root <none> and password file <none>
I20260923 14:48:10.400197 12783 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:10.400266 12783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:10.404088 12783 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:48:10.406193 12803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:10.407460 12783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:48:10.407608 12783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "5e1fd031be08463cbda4dfc74eaef121"
format_stamp: "Formatted at 2026-09-23 14:48:09 on dist-test-slave-wjf2"
server_key: "15ae710df0904708cfd60deddd3ae378"
server_key_iv: "e35ec314212e874ea89062d17ba477b8"
server_key_version: "encryptionkey@0"
I20260923 14:48:10.408197 12783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:10.426434 12783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:10.429587 12783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:10.429781 12783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:10.430358 12783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:10.431152 12783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:10.431219 12783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:10.431288 12783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:10.431329 12783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:10.439942 12783 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:45189
I20260923 14:48:10.440008 12916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:45189 every 8 connection(s)
I20260923 14:48:10.440954 12783 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:48:10.441769 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 12783
I20260923 14:48:10.442143 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:41461
--local_ip_for_outbound_sockets=127.30.2.195
--tserver_master_addrs=127.30.2.254:33037
--webserver_port=35313
--webserver_interface=127.30.2.195
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:48:10 dist-test-slave-wjf2 krb5kdc[12009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174890, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:10.455572 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.442822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:54191 (local address 127.30.2.254:33037)
0923 14:48:10.442986 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:10.442991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:10.443947 (+ 956us) server_negotiation.cc:407] Connection header received
0923 14:48:10.445336 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:10.445343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:10.445411 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:10.445522 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:10.446670 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.447609 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.448833 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.449277 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.449414 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.451864 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:10.451872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:10.451875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:10.451908 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:10.453584 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:10.454047 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:10.454050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:10.454052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:10.454137 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:10.454431 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:10.454434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:10.454436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:10.454652 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:10.454748 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:10.455254 (+ 506us) server_negotiation.cc:299] Negotiation successful
0923 14:48:10.455385 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 14:48:10.456224 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.443236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:54191)
0923 14:48:10.443762 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:10.443834 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:10.445123 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:10.445669 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:10.445675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:10.446030 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:10.446490 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.446503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.447764 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.447767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.448707 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.448717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.449425 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.449427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.449575 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.450179 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:10.450195 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:10.451664 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:10.453699 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:10.453704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:10.453715 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:10.453951 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:10.454224 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:10.454226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:10.454227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:10.454346 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:10.454749 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:10.454754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:10.455009 (+ 255us) client_negotiation.cc:769] Sending connection context
0923 14:48:10.455199 (+ 190us) client_negotiation.cc:240] Negotiation successful
0923 14:48:10.455437 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260923 14:48:10.457361 12917 heartbeater.cc:347] Connected to a master server at 127.30.2.254:33037
I20260923 14:48:10.457612 12917 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:10.458155 12917 heartbeater.cc:511] Master 127.30.2.254:33037 requested a full tablet report, sending...
I20260923 14:48:10.459254 12591 ts_manager.cc:194] Registered new tserver with Master: 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189)
I20260923 14:48:10.459887 12591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:54191
W20260923 14:48:10.557250 12919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:10.557482 12919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:10.557557 12919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:10.561633 12919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:10.561681 12919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:10.561769 12919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:48:10.566615 12919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:41461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=35313
--tserver_master_addrs=127.30.2.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:10.567595 12919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:10.568604 12919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:10.577927 12925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:10.577932 12926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:10.577939 12928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:10.578474 12919 server_base.cc:1034] running on GCE node
I20260923 14:48:10.578888 12919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:10.579596 12919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:10.580817 12919 hybrid_clock.cc:648] HybridClock initialized: now 1790174890580767 us; error 59 us; skew 500 ppm
Sep 23 14:48:10 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174890, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:10.583602 12919 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:48:10.584604 12919 webserver.cc:460] Webserver started at http://127.30.2.195:35313/ using document root <none> and password file <none>
I20260923 14:48:10.585201 12919 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:10.585320 12919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:10.589486 12919 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260923 14:48:10.591591 12935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:10.592830 12919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:48:10.593020 12919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "abefccf81418440b8b4bc6edb53e0c14"
format_stamp: "Formatted at 2026-09-23 14:48:09 on dist-test-slave-wjf2"
server_key: "f805607a0e9c6ad5a7a4c3fc07db675a"
server_key_iv: "5ca9b8787e010195716e1677df883db2"
server_key_version: "encryptionkey@0"
I20260923 14:48:10.593586 12919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:10.625874 12919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:10.629835 12919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:10.630069 12919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:10.630739 12919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:10.631636 12919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:10.631712 12919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:10.631811 12919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:10.631858 12919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:10.641310 12919 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:41461
I20260923 14:48:10.641330 13048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:41461 every 8 connection(s)
I20260923 14:48:10.642380 12919 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:48:10.649207 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 12919
Sep 23 14:48:10 dist-test-slave-wjf2 krb5kdc[12009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174890, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:10.657735 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.644359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:47705 (local address 127.30.2.254:33037)
0923 14:48:10.644549 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:10.644555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:10.645524 (+ 969us) server_negotiation.cc:407] Connection header received
0923 14:48:10.646849 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:10.646852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:10.646903 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:10.646987 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:10.648243 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.648789 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.650393 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.650735 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.650861 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.653541 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:10.653545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:10.653547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:10.653582 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:10.655383 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:10.656075 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:10.656078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:10.656079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:10.656153 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:10.656572 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:10.656574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:10.656576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:10.656735 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:10.656810 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:10.657434 (+ 624us) server_negotiation.cc:299] Negotiation successful
0923 14:48:10.657548 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 14:48:10.658627 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.644865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:47705)
0923 14:48:10.645362 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:10.645404 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:10.646622 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:10.647156 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:10.647163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:10.647517 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:10.648080 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.648096 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.648931 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.648936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.650234 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.650246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.650883 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.650886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.651036 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.651663 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:10.651679 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:10.653324 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:10.655521 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:10.655528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:10.655544 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:10.655936 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:10.656251 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:10.656254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:10.656256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:10.656453 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:10.656840 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:10.656847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:10.657167 (+ 320us) client_negotiation.cc:769] Sending connection context
0923 14:48:10.657329 (+ 162us) client_negotiation.cc:240] Negotiation successful
0923 14:48:10.657680 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":130,"threads_started":1}
I20260923 14:48:10.658784 13053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.649903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.30.2.254:33037)
0923 14:48:10.650176 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:10.650181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:10.650242 (+ 61us) server_negotiation.cc:407] Connection header received
0923 14:48:10.650354 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:10.650357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:10.650417 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:10.650487 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:10.651418 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.652081 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.653053 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.653390 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.653490 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.655298 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:10.655305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:10.655310 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:10.655360 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:10.657037 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:10.657462 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:10.657465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:10.657466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:10.657551 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:10.657903 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:10.657909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:10.657911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:10.658101 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:10.658308 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:10.658515 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 14:48:10.658630 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
I20260923 14:48:10.659861 13049 heartbeater.cc:347] Connected to a master server at 127.30.2.254:33037
I20260923 14:48:10.660120 13049 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:10.660724 13049 heartbeater.cc:511] Master 127.30.2.254:33037 requested a full tablet report, sending...
I20260923 14:48:10.661697 12591 ts_manager.cc:194] Registered new tserver with Master: abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461)
I20260923 14:48:10.662230 12591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:47705
I20260923 14:48:10.672178 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:48:10.676159 13053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.672755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.30.2.254:33037)
0923 14:48:10.672865 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:10.672869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:10.673039 (+ 170us) server_negotiation.cc:407] Connection header received
0923 14:48:10.673172 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:10.673175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:10.673259 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:10.673339 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:48:10.673594 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.674072 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.674879 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.675109 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.675184 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.675441 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:48:10.675840 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:48:10.675942 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:10.676016 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 14:48:10.676066 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:48:10.676767 12591 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:48:10.680045 12591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:48:10.687768 12713 tablet_service.cc:1467] Processing CreateTablet for tablet 3e69a97ce2ae4c5cac259582d28d229d (DEFAULT_TABLE table=test-table [id=c0b68f08f14346a5847fcca26a660dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:10.688470 12713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e69a97ce2ae4c5cac259582d28d229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:10.691354 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.686546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:41461 (local address 127.0.0.1:55556)
0923 14:48:10.686666 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:10.686681 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:10.686772 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:10.687851 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:10.687854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:10.687872 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:10.687975 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.687982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.689068 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.689075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.690128 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.690137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.690937 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.690940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.691034 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.691071 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 14:48:10.691155 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 14:48:10.691233 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 14:48:10.691587 12671 tablet_bootstrap.cc:492] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: Bootstrap starting.
I20260923 14:48:10.691592 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.686974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.30.2.195:41461)
0923 14:48:10.687424 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:10.687433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:10.687453 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:48:10.687520 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:10.687523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:10.687681 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:10.687872 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:10.688074 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.688930 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.690253 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.690804 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.690936 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.691093 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:10.691182 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 14:48:10.691267 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":131,"threads_started":1}
I20260923 14:48:10.692238 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.686811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:45189 (local address 127.0.0.1:33242)
0923 14:48:10.687079 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:10.687095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:10.687178 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:10.687942 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:10.687945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:10.687965 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:10.688053 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.688059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.689565 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.689568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.690655 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.690665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.691859 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.691863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.691948 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.691999 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 14:48:10.692054 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 14:48:10.692130 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":94,"threads_started":1}
I20260923 14:48:10.692580 12671 tablet_bootstrap.cc:654] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:10.693384 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.687054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.30.2.194:45189)
0923 14:48:10.687430 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:10.687442 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:10.687476 (+ 34us) server_negotiation.cc:407] Connection header received
0923 14:48:10.687551 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:10.687556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:10.687745 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:10.688198 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:10.688360 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.689400 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.690823 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.691595 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.692882 (+ 1287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.693077 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:10.693132 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 14:48:10.693226 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":105,"threads_started":1}
I20260923 14:48:10.694638 12671 tablet_bootstrap.cc:492] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: No bootstrap required, opened a new log
I20260923 14:48:10.694746 12671 ts_tablet_manager.cc:1397] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 14:48:10.695269 12671 raft_consensus.cc:348] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:10.695403 12671 raft_consensus.cc:374] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:10.695453 12671 raft_consensus.cc:729] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7889a3e96284cfba0427850c5e08124, State: Initialized, Role: FOLLOWER
I20260923 14:48:10.695657 12671 consensus_queue.cc:260] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:10.695242 12983 tablet_service.cc:1467] Processing CreateTablet for tablet 3e69a97ce2ae4c5cac259582d28d229d (DEFAULT_TABLE table=test-table [id=c0b68f08f14346a5847fcca26a660dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:10.696036 12671 ts_tablet_manager.cc:1428] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:48:10.696388 12983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e69a97ce2ae4c5cac259582d28d229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:10.696125 12851 tablet_service.cc:1467] Processing CreateTablet for tablet 3e69a97ce2ae4c5cac259582d28d229d (DEFAULT_TABLE table=test-table [id=c0b68f08f14346a5847fcca26a660dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:10.697110 12851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e69a97ce2ae4c5cac259582d28d229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:10.705117 13063 tablet_bootstrap.cc:492] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121: Bootstrap starting.
I20260923 14:48:10.707345 13063 tablet_bootstrap.cc:654] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:10.707845 13062 tablet_bootstrap.cc:492] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14: Bootstrap starting.
I20260923 14:48:10.708369 13063 log.cc:824] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:10.710224 13062 tablet_bootstrap.cc:654] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:10.710316 13063 tablet_bootstrap.cc:492] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121: No bootstrap required, opened a new log
I20260923 14:48:10.710507 13063 ts_tablet_manager.cc:1397] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:48:10.711012 13062 log.cc:824] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:10.712944 13062 tablet_bootstrap.cc:492] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14: No bootstrap required, opened a new log
I20260923 14:48:10.713135 13062 ts_tablet_manager.cc:1397] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:48:10.713646 13063 raft_consensus.cc:348] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:10.713892 13063 raft_consensus.cc:374] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:10.713959 13063 raft_consensus.cc:729] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1fd031be08463cbda4dfc74eaef121, State: Initialized, Role: FOLLOWER
I20260923 14:48:10.714444 13063 consensus_queue.cc:260] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:10.715167 12917 heartbeater.cc:503] Master 127.30.2.254:33037 was elected leader, sending a full tablet report...
I20260923 14:48:10.715425 13063 ts_tablet_manager.cc:1428] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 14:48:10.716583 13062 raft_consensus.cc:348] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:10.716800 13062 raft_consensus.cc:374] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:10.716864 13062 raft_consensus.cc:729] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abefccf81418440b8b4bc6edb53e0c14, State: Initialized, Role: FOLLOWER
I20260923 14:48:10.717276 13062 consensus_queue.cc:260] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:10.717917 13049 heartbeater.cc:503] Master 127.30.2.254:33037 was elected leader, sending a full tablet report...
I20260923 14:48:10.718214 13062 ts_tablet_manager.cc:1428] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260923 14:48:10.724206 12785 raft_consensus.cc:482] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:48:10.724407 12785 raft_consensus.cc:504] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:10.725467 12785 leader_election.cc:288] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461), 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189)
I20260923 14:48:10.729615 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.725851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:58371 (local address 127.30.2.194:45189)
0923 14:48:10.725983 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:10.725987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:10.726188 (+ 201us) server_negotiation.cc:407] Connection header received
0923 14:48:10.726283 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:10.726286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:10.726330 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:10.726411 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:10.726729 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.727281 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.728658 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.729043 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.729176 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.729226 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:10.729430 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 14:48:10.729500 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:48:10.729753 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.725615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:41461 (local address 127.30.2.193:43687)
0923 14:48:10.725796 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:10.725811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:10.725908 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:10.726256 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:10.726261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:10.726282 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:10.726419 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.726426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.727386 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.727391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.728489 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.728499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.729298 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.729302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.729409 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.729446 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 14:48:10.729526 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 14:48:10.729601 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":80}
I20260923 14:48:10.729913 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.725698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:43687 (local address 127.30.2.195:41461)
0923 14:48:10.725861 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:10.725869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:10.725887 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:48:10.726045 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:10.726056 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:10.726149 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:10.726233 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:10.726513 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.727262 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.728613 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.729167 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.729308 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.729360 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:10.729703 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 14:48:10.729774 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 14:48:10.729645 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.725759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:45189 (local address 127.30.2.193:58371)
0923 14:48:10.726064 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:10.726078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:10.726193 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:10.726458 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:10.726461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:10.726482 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:10.726610 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.726618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.727391 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.727395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.728508 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:10.728518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.729177 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.729181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:10.729289 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.729323 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 14:48:10.729414 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 14:48:10.729487 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260923 14:48:10.730501 13003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e69a97ce2ae4c5cac259582d28d229d" candidate_uuid: "c7889a3e96284cfba0427850c5e08124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abefccf81418440b8b4bc6edb53e0c14" is_pre_election: true
I20260923 14:48:10.730607 12871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e69a97ce2ae4c5cac259582d28d229d" candidate_uuid: "c7889a3e96284cfba0427850c5e08124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1fd031be08463cbda4dfc74eaef121" is_pre_election: true
I20260923 14:48:10.730782 13003 raft_consensus.cc:2454] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7889a3e96284cfba0427850c5e08124 in term 0.
I20260923 14:48:10.730926 12871 raft_consensus.cc:2454] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7889a3e96284cfba0427850c5e08124 in term 0.
I20260923 14:48:10.731305 12667 leader_election.cc:302] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abefccf81418440b8b4bc6edb53e0c14, c7889a3e96284cfba0427850c5e08124; no voters:
I20260923 14:48:10.731613 12785 raft_consensus.cc:2781] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:48:10.731699 12785 raft_consensus.cc:482] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:48:10.731746 12785 raft_consensus.cc:3037] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:10.732771 12785 raft_consensus.cc:504] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:10.733100 12785 leader_election.cc:288] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 1 election: Requested vote from peers abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461), 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189)
I20260923 14:48:10.733446 13003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e69a97ce2ae4c5cac259582d28d229d" candidate_uuid: "c7889a3e96284cfba0427850c5e08124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abefccf81418440b8b4bc6edb53e0c14"
I20260923 14:48:10.733501 12871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e69a97ce2ae4c5cac259582d28d229d" candidate_uuid: "c7889a3e96284cfba0427850c5e08124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1fd031be08463cbda4dfc74eaef121"
I20260923 14:48:10.733554 13003 raft_consensus.cc:3037] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:10.733649 12871 raft_consensus.cc:3037] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:10.734629 13003 raft_consensus.cc:2454] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7889a3e96284cfba0427850c5e08124 in term 1.
I20260923 14:48:10.734807 12871 raft_consensus.cc:2454] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7889a3e96284cfba0427850c5e08124 in term 1.
I20260923 14:48:10.734956 12667 leader_election.cc:302] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abefccf81418440b8b4bc6edb53e0c14, c7889a3e96284cfba0427850c5e08124; no voters:
I20260923 14:48:10.735126 12785 raft_consensus.cc:2781] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:48:10.735368 12785 raft_consensus.cc:686] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 LEADER]: Becoming Leader. State: Replica: c7889a3e96284cfba0427850c5e08124, State: Running, Role: LEADER
I20260923 14:48:10.735620 12785 consensus_queue.cc:237] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:10.738350 12591 catalog_manager.cc:5495] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7889a3e96284cfba0427850c5e08124 (127.30.2.193). New cstate: current_term: 1 leader_uuid: "c7889a3e96284cfba0427850c5e08124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } health_report { overall_health: HEALTHY } } }
W20260923 14:48:10.748837 12780 tablet.cc:2369] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:48:10.760998 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.757400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.30.2.193:42783)
0923 14:48:10.757530 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:10.757536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:10.757575 (+ 39us) server_negotiation.cc:407] Connection header received
0923 14:48:10.757774 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:10.757777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:10.757830 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:10.757935 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:48:10.758177 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.758695 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.759534 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:10.759822 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:10.759940 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:10.760214 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:48:10.760623 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:48:10.760740 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:10.760799 (+ 59us) server_negotiation.cc:299] Negotiation successful
0923 14:48:10.760857 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:48:10.764760 13003 raft_consensus.cc:1261] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14 [term 1 FOLLOWER]: Refusing update from remote peer c7889a3e96284cfba0427850c5e08124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:10.764760 12871 raft_consensus.cc:1261] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121 [term 1 FOLLOWER]: Refusing update from remote peer c7889a3e96284cfba0427850c5e08124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:10.765545 13069 consensus_queue.cc:1035] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:48:10.765743 12785 consensus_queue.cc:1035] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:48:10.781849 12591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:48:10.782004 12591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:48:10.784385 12591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Sending DeleteTablet for 3 replicas of tablet 3e69a97ce2ae4c5cac259582d28d229d
I20260923 14:48:10.785215 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 12629
I20260923 14:48:10.785197 12983 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e69a97ce2ae4c5cac259582d28d229d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:10 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:55556
I20260923 14:48:10.785387 12851 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e69a97ce2ae4c5cac259582d28d229d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:10 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:33242
I20260923 14:48:10.786158 13081 tablet_replica.cc:330] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14: stopping tablet replica
I20260923 14:48:10.786402 13082 tablet_replica.cc:330] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121: stopping tablet replica
I20260923 14:48:10.786485 13081 raft_consensus.cc:2229] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:48:10.786752 13082 raft_consensus.cc:2229] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:48:10.786819 13081 raft_consensus.cc:2258] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:10.787027 13082 raft_consensus.cc:2258] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:10.788575 13082 ts_tablet_manager.cc:1905] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:10.789752 13081 ts_tablet_manager.cc:1905] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:10.791916 13081 ts_tablet_manager.cc:1918] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:10.792044 13081 log.cc:1196] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/3e69a97ce2ae4c5cac259582d28d229d
I20260923 14:48:10.792090 13082 ts_tablet_manager.cc:1918] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:10.792200 13082 log.cc:1196] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/3e69a97ce2ae4c5cac259582d28d229d
I20260923 14:48:10.792395 13081 ts_tablet_manager.cc:1939] T 3e69a97ce2ae4c5cac259582d28d229d P abefccf81418440b8b4bc6edb53e0c14: Deleting consensus metadata
I20260923 14:48:10.792577 13082 ts_tablet_manager.cc:1939] T 3e69a97ce2ae4c5cac259582d28d229d P 5e1fd031be08463cbda4dfc74eaef121: Deleting consensus metadata
I20260923 14:48:10.793512 12577 catalog_manager.cc:4841] TS abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461): tablet 3e69a97ce2ae4c5cac259582d28d229d (table test-table [id=c0b68f08f14346a5847fcca26a660dac]) successfully deleted
I20260923 14:48:10.793751 12575 catalog_manager.cc:4841] TS 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189): tablet 3e69a97ce2ae4c5cac259582d28d229d (table test-table [id=c0b68f08f14346a5847fcca26a660dac]) successfully deleted
W20260923 14:48:10.794420 12575 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:42783 (error 108)
I20260923 14:48:10.794731 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 12783
I20260923 14:48:10.796303 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:10.795216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:42783 (local address 127.0.0.1:36448)
0923 14:48:10.796161 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:10.796233 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":843}
W20260923 14:48:10.796452 12575 catalog_manager.cc:4584] TS c7889a3e96284cfba0427850c5e08124 (127.30.2.193:42783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e69a97ce2ae4c5cac259582d28d229d: Network error: Client connection negotiation failed: client connection to 127.30.2.193:42783: connect: Connection refused (error 111)
I20260923 14:48:10.801844 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 12919
I20260923 14:48:10.808072 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 12558
I20260923 14:48:10.814954 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:33037
--webserver_interface=127.30.2.254
--webserver_port=44579
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:33037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:10.927811 13084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:10.928061 13084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:10.928141 13084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:10.932152 13084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:48:10.932199 13084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:10.932217 13084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:10.932233 13084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:48:10.932286 13084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:48:10.937475 13084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:33037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=44579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:10.938483 13084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:10.939565 13084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:10.948762 13090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:10.948833 13084 server_base.cc:1034] running on GCE node
W20260923 14:48:10.948732 13092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:10.948730 13089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:10.949517 13084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:10.950503 13084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:10.951659 13084 hybrid_clock.cc:648] HybridClock initialized: now 1790174890951623 us; error 47 us; skew 500 ppm
Sep 23 14:48:10 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174890, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:10.954284 13084 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:48:10.955051 13084 webserver.cc:460] Webserver started at http://127.30.2.254:44579/ using document root <none> and password file <none>
I20260923 14:48:10.955564 13084 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:10.955644 13084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:10.959297 13084 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:48:10.961323 13099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:10.962498 13084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:48:10.962618 13084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "f61def0074ec4acfb053f1d3c37444b5"
format_stamp: "Formatted at 2026-09-23 14:48:08 on dist-test-slave-wjf2"
server_key: "9f1cfbb6f9665de4213a83df5061c83f"
server_key_iv: "756f7d756de30f7ba4080a911a4dc901"
server_key_version: "encryptionkey@0"
I20260923 14:48:10.963119 13084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:10.983927 13084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:10.986629 13084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:10.986851 13084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:10.993559 13084 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:33037
I20260923 14:48:10.993839 13151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:33037 every 8 connection(s)
I20260923 14:48:10.994602 13084 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:48:10.998893 13152 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:48:11.000543 13152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Bootstrap starting.
I20260923 14:48:11.001170 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 13084
I20260923 14:48:11.001600 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:42783
--local_ip_for_outbound_sockets=127.30.2.193
--tserver_master_addrs=127.30.2.254:33037
--webserver_port=36201
--webserver_interface=127.30.2.193
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:48:11.017642 13152 log.cc:824] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:11.029042 13152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:48:11.029409 13152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5: Bootstrap complete.
I20260923 14:48:11.033479 13152 raft_consensus.cc:348] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:11.033890 13152 raft_consensus.cc:729] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61def0074ec4acfb053f1d3c37444b5, State: Initialized, Role: FOLLOWER
I20260923 14:48:11.034425 13152 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:11.034579 13152 raft_consensus.cc:388] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:48:11.034670 13152 raft_consensus.cc:482] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:48:11.034794 13152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 2 FOLLOWER]: Advancing to term 3
I20260923 14:48:11.036718 13152 raft_consensus.cc:504] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:11.037051 13152 leader_election.cc:302] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f61def0074ec4acfb053f1d3c37444b5; no voters:
I20260923 14:48:11.037389 13152 leader_election.cc:288] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 14:48:11.037483 13156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 14:48:11.037676 13156 raft_consensus.cc:686] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [term 3 LEADER]: Becoming Leader. State: Replica: f61def0074ec4acfb053f1d3c37444b5, State: Running, Role: LEADER
I20260923 14:48:11.037974 13156 consensus_queue.cc:237] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } }
I20260923 14:48:11.038671 13152 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:48:11.039310 13157 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f61def0074ec4acfb053f1d3c37444b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } } }
I20260923 14:48:11.039709 13157 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:11.039619 13158 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61def0074ec4acfb053f1d3c37444b5. Latest consensus state: current_term: 3 leader_uuid: "f61def0074ec4acfb053f1d3c37444b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61def0074ec4acfb053f1d3c37444b5" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33037 } } }
I20260923 14:48:11.040076 13158 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61def0074ec4acfb053f1d3c37444b5 [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:11.040362 13166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:48:11.045506 13166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:48:11.046634 13166 catalog_manager.cc:1259] Loaded cluster ID: f7183d6b4f1444d59688d1f83e8c5096
I20260923 14:48:11.046723 13166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 14:48:11.047730 13166 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2f3753d9ad google::LogMessage::Fail() at ??:0
@ 0x7f2f3754088a google::LogMessage::SendToLog() at ??:0
@ 0x7f2f3753d4de google::LogMessage::Flush() at ??:0
@ 0x7f2f3753ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2f41e7cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f2f41e7dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2f41e7a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f2f41ebe640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2f3978f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2f3978fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f2f397913f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2f39780512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2f3adc46db start_thread at ??:0
@ 0x7f2f3603b61f clone at ??:0
W20260923 14:48:11.142759 13154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:11.143011 13154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:11.143055 13154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:11.147087 13154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:11.147138 13154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:11.147194 13154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:48:11.152379 13154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:42783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=36201
--tserver_master_addrs=127.30.2.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:11.153445 13154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:11.154470 13154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:11.163913 13193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:11.163954 13194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:11.164268 13154 server_base.cc:1034] running on GCE node
W20260923 14:48:11.164325 13196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:11.164726 13154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:11.165530 13154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:11.166739 13154 hybrid_clock.cc:648] HybridClock initialized: now 1790174891166712 us; error 31 us; skew 500 ppm
Sep 23 14:48:11 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174891, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:11.170156 13154 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:48:11.171226 13154 webserver.cc:460] Webserver started at http://127.30.2.193:36201/ using document root <none> and password file <none>
I20260923 14:48:11.171927 13154 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:11.172032 13154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:11.176892 13154 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:48:11.179503 13203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:11.181102 13154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:48:11.181227 13154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "c7889a3e96284cfba0427850c5e08124"
format_stamp: "Formatted at 2026-09-23 14:48:08 on dist-test-slave-wjf2"
server_key: "f6192e5bbfc90f16e5649a79996bf667"
server_key_iv: "7ffefb2df733a38459c934d5f98a7974"
server_key_version: "encryptionkey@0"
I20260923 14:48:11.181905 13154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:11.213070 13154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:11.216915 13154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:11.217129 13154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:11.217875 13154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:11.219228 13210 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 14:48:11.224480 13154 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 14:48:11.224556 13154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260923 14:48:11.224625 13154 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 14:48:11.226866 13154 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 14:48:11.226922 13154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260923 14:48:11.227254 13210 tablet_bootstrap.cc:492] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: Bootstrap starting.
I20260923 14:48:11.238099 13154 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:42783
I20260923 14:48:11.238610 13317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:42783 every 8 connection(s)
I20260923 14:48:11.239300 13154 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:48:11.248783 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 13154
I20260923 14:48:11.249293 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:45189
--local_ip_for_outbound_sockets=127.30.2.194
--tserver_master_addrs=127.30.2.254:33037
--webserver_port=46339
--webserver_interface=127.30.2.194
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:48:11.253743 13210 log.cc:824] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:11.261145 13210 tablet_bootstrap.cc:492] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:48:11.261565 13210 tablet_bootstrap.cc:492] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: Bootstrap complete.
I20260923 14:48:11.262676 13210 ts_tablet_manager.cc:1397] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: Time spent bootstrapping tablet: real 0.036s user 0.022s sys 0.004s
I20260923 14:48:11.265393 13210 raft_consensus.cc:348] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:11.265947 13210 raft_consensus.cc:729] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7889a3e96284cfba0427850c5e08124, State: Initialized, Role: FOLLOWER
I20260923 14:48:11.266465 13210 consensus_queue.cc:260] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:11.267525 13210 ts_tablet_manager.cc:1428] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260923 14:48:11.271133 13320 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:48:11.242247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:49883)
0923 14:48:11.242887 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:11.242977 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:11.244105 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:11.270388 (+ 26283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":149,"threads_started":1}
W20260923 14:48:11.271626 13318 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:33037 (0 consecutive failures): Network error: Failed to ping master at 127.30.2.254:33037: Client connection negotiation failed: client connection to 127.30.2.254:33037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 14:48:11.272354 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:11.272113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:43685)
0923 14:48:11.272240 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:11.272301 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 14:48:11.272979 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:11.272772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:40741)
0923 14:48:11.272880 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:11.272931 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 14:48:11.273134 13318 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:48:11.366468 13321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:11.366740 13321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:11.366816 13321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:11.370864 13321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:11.370942 13321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:11.371031 13321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:48:11.375954 13321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:45189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=46339
--tserver_master_addrs=127.30.2.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:11.376909 13321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:11.377915 13321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:11.387531 13331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:11.387509 13321 server_base.cc:1034] running on GCE node
W20260923 14:48:11.387470 13329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:11.387493 13328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:11.388173 13321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:11.388775 13321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:11.389976 13321 hybrid_clock.cc:648] HybridClock initialized: now 1790174891389966 us; error 37 us; skew 500 ppm
Sep 23 14:48:11 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174891, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:11.392719 13321 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:48:11.393522 13321 webserver.cc:460] Webserver started at http://127.30.2.194:46339/ using document root <none> and password file <none>
I20260923 14:48:11.394101 13321 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:11.394189 13321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:11.397858 13321 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260923 14:48:11.399821 13338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:11.401016 13321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:48:11.401150 13321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "5e1fd031be08463cbda4dfc74eaef121"
format_stamp: "Formatted at 2026-09-23 14:48:09 on dist-test-slave-wjf2"
server_key: "15ae710df0904708cfd60deddd3ae378"
server_key_iv: "e35ec314212e874ea89062d17ba477b8"
server_key_version: "encryptionkey@0"
I20260923 14:48:11.401656 13321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:11.424587 13321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:11.427992 13321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:11.428220 13321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:11.428821 13321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:11.429643 13321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:11.429715 13321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:11.429780 13321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:11.429831 13321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:11.438522 13321 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:45189
I20260923 14:48:11.438583 13451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:45189 every 8 connection(s)
I20260923 14:48:11.439512 13321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:48:11.443480 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:11.441682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:52615)
0923 14:48:11.442215 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:11.442650 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":110,"threads_started":1}
W20260923 14:48:11.443984 13452 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:33037 (0 consecutive failures): Network error: Failed to ping master at 127.30.2.254:33037: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
I20260923 14:48:11.444648 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:11.444380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:57739)
0923 14:48:11.444522 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:11.444576 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 14:48:11.445300 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:11.445101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:44843)
0923 14:48:11.445209 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:11.445252 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 14:48:11.445518 13452 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 14:48:11.448642 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 13321
I20260923 14:48:11.449069 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:41461
--local_ip_for_outbound_sockets=127.30.2.195
--tserver_master_addrs=127.30.2.254:33037
--webserver_port=35313
--webserver_interface=127.30.2.195
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:11.557567 13455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:11.557785 13455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:11.557824 13455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:11.561661 13455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:11.561707 13455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:11.561760 13455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:48:11.566452 13455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:41461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=35313
--tserver_master_addrs=127.30.2.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:11.567358 13455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:11.568277 13455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:11.577560 13460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:11.577634 13455 server_base.cc:1034] running on GCE node
W20260923 14:48:11.577668 13463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:11.577778 13461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:11.578137 13455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:11.578724 13455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:11.579860 13455 hybrid_clock.cc:648] HybridClock initialized: now 1790174891579842 us; error 29 us; skew 500 ppm
Sep 23 14:48:11 dist-test-slave-wjf2 krb5kdc[12009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174891, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:11.582507 13455 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:48:11.583492 13455 webserver.cc:460] Webserver started at http://127.30.2.195:35313/ using document root <none> and password file <none>
I20260923 14:48:11.584210 13455 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:11.584334 13455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:11.588449 13455 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:48:11.590708 13470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:11.592085 13455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:48:11.592234 13455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "abefccf81418440b8b4bc6edb53e0c14"
format_stamp: "Formatted at 2026-09-23 14:48:09 on dist-test-slave-wjf2"
server_key: "f805607a0e9c6ad5a7a4c3fc07db675a"
server_key_iv: "5ca9b8787e010195716e1677df883db2"
server_key_version: "encryptionkey@0"
I20260923 14:48:11.592828 13455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:11.611498 13455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:11.614748 13455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:11.614967 13455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:11.615641 13455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:11.616603 13455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:11.616690 13455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:11.616772 13455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:11.616825 13455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:11.627434 13455 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:41461
I20260923 14:48:11.627480 13583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:41461 every 8 connection(s)
I20260923 14:48:11.628690 13455 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:48:11.632609 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:11.630904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:35013)
0923 14:48:11.631414 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:11.631915 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":129,"threads_started":1}
W20260923 14:48:11.633059 13584 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:33037 (0 consecutive failures): Network error: Failed to ping master at 127.30.2.254:33037: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
I20260923 14:48:11.633594 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:11.633378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:41211)
0923 14:48:11.633477 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:11.633544 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 14:48:11.634171 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:11.633983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:37289)
0923 14:48:11.634087 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:11.634125 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 14:48:11.634375 13584 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 14:48:11.635169 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 13455
I20260923 14:48:12.274121 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:12.273685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:33043)
0923 14:48:12.273954 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:12.274015 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
I20260923 14:48:12.446486 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:12.446066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:49885)
0923 14:48:12.446328 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:12.446384 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":67,"threads_started":1}
I20260923 14:48:12.635363 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:12.634895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:55881)
0923 14:48:12.635180 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:12.635254 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260923 14:48:13.056465 13591 raft_consensus.cc:482] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:48:13.056690 13591 raft_consensus.cc:504] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:13.057907 13591 leader_election.cc:288] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461), 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189)
Sep 23 14:48:13 dist-test-slave-wjf2 krb5kdc[12009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174891, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.194@KRBTEST.COM
Sep 23 14:48:13 dist-test-slave-wjf2 krb5kdc[12009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174891, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.195@KRBTEST.COM
I20260923 14:48:13.071210 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:13.058177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:45189 (local address 127.30.2.193:55451)
0923 14:48:13.058506 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:13.058525 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:13.058657 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:13.060226 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:13.060232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:13.060816 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:13.061324 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:13.061343 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:13.062597 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:13.062606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:13.063541 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:13.063551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:13.064321 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:13.064325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:13.064506 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:13.065189 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:13.065220 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:13.066869 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:13.068965 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:13.068971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:13.068984 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:13.069223 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:13.069564 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:13.069567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:13.069569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:13.069700 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:13.070285 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:13.070291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:13.070505 (+ 214us) client_negotiation.cc:769] Sending connection context
0923 14:48:13.070692 (+ 187us) client_negotiation.cc:240] Negotiation successful
0923 14:48:13.071029 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":159,"thread_start_us":87,"threads_started":1}
I20260923 14:48:13.071262 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:13.058384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:55451 (local address 127.30.2.194:45189)
0923 14:48:13.058808 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:13.058820 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:13.058855 (+ 35us) server_negotiation.cc:407] Connection header received
0923 14:48:13.059499 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:13.059504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:13.059892 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:13.060180 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:13.061479 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:13.062452 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:13.063661 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:13.064186 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:13.064368 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:13.067084 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:13.067090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:13.067100 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:13.067125 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:13.068845 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:13.069336 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:13.069342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:13.069354 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:13.069459 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:13.069792 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:13.069797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:13.069799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:13.070160 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:13.070295 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:13.070840 (+ 545us) server_negotiation.cc:299] Negotiation successful
0923 14:48:13.071101 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":75,"threads_started":1}
I20260923 14:48:13.071305 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:13.058390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:45131 (local address 127.30.2.195:41461)
0923 14:48:13.058814 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:13.058834 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:13.058864 (+ 30us) server_negotiation.cc:407] Connection header received
0923 14:48:13.059515 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:13.059520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:13.059869 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:13.060181 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:13.061519 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:13.062468 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:13.063663 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:13.064193 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:13.064368 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:13.067265 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:13.067270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:13.067280 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:13.067305 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:13.069100 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:13.069560 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:13.069565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:13.069570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:13.069681 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:13.069982 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:13.069985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:13.069987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:13.070328 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:13.070463 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:13.070850 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 14:48:13.071119 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":59,"threads_started":1}
I20260923 14:48:13.072062 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:13.058223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:41461 (local address 127.30.2.193:45131)
0923 14:48:13.058573 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:13.058587 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:13.058685 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:13.060340 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:13.060343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:13.060816 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:13.061324 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:13.061340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:13.062601 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:13.062604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:13.063539 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:13.063549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:13.064317 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:13.064320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:13.064502 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:13.065250 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:13.065263 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:13.067082 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:13.069238 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:13.069241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:13.069244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:13.069437 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:13.069788 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:13.069791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:13.069793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:13.069876 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:13.070474 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:13.070480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:13.070574 (+ 94us) client_negotiation.cc:769] Sending connection context
0923 14:48:13.070701 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 14:48:13.071177 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":226,"thread_start_us":85,"threads_started":1}
W20260923 14:48:13.074177 13204 leader_election.cc:341] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189): Not found: Tablet not found: 3e69a97ce2ae4c5cac259582d28d229d
W20260923 14:48:13.074362 13206 leader_election.cc:341] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461): Not found: Tablet not found: 3e69a97ce2ae4c5cac259582d28d229d
I20260923 14:48:13.074463 13206 leader_election.cc:302] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7889a3e96284cfba0427850c5e08124; no voters: 5e1fd031be08463cbda4dfc74eaef121, abefccf81418440b8b4bc6edb53e0c14
I20260923 14:48:13.074702 13591 raft_consensus.cc:2726] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:48:13.275282 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:13.274953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:46157)
0923 14:48:13.275133 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:13.275194 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 14:48:13.447683 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:13.447255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:48659)
0923 14:48:13.447521 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:13.447576 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260923 14:48:13.636483 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:13.636105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:40383)
0923 14:48:13.636339 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:13.636390 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260923 14:48:14.276530 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:14.276121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:47927)
0923 14:48:14.276372 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:14.276425 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260923 14:48:14.448907 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:14.448496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:43473)
0923 14:48:14.448762 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:14.448817 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260923 14:48:14.637693 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:14.637254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:34169)
0923 14:48:14.637535 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:14.637587 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
I20260923 14:48:14.844609 13601 raft_consensus.cc:482] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:48:14.844790 13601 raft_consensus.cc:504] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:14.845250 13601 leader_election.cc:288] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461), 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189)
W20260923 14:48:14.846145 13204 leader_election.cc:341] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189): Not found: Tablet not found: 3e69a97ce2ae4c5cac259582d28d229d
W20260923 14:48:14.846243 13206 leader_election.cc:341] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461): Not found: Tablet not found: 3e69a97ce2ae4c5cac259582d28d229d
I20260923 14:48:14.846298 13206 leader_election.cc:302] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7889a3e96284cfba0427850c5e08124; no voters: 5e1fd031be08463cbda4dfc74eaef121, abefccf81418440b8b4bc6edb53e0c14
I20260923 14:48:14.846439 13601 raft_consensus.cc:2726] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:48:15.277792 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:15.277287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:44067)
0923 14:48:15.277603 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:15.277667 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":94,"threads_started":1}
I20260923 14:48:15.450011 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:15.449620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:53713)
0923 14:48:15.449864 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:15.449916 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260923 14:48:15.639070 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:15.638528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:60435)
0923 14:48:15.638850 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:15.638926 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":75,"threads_started":1}
I20260923 14:48:16.279263 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:16.278739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:53507)
0923 14:48:16.279077 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:16.279129 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":98,"threads_started":1}
I20260923 14:48:16.451326 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:16.450839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:45791)
0923 14:48:16.451131 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:16.451224 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":112,"threads_started":1}
I20260923 14:48:16.640352 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:16.639942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:49491)
0923 14:48:16.640208 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:16.640263 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260923 14:48:17.077767 13608 raft_consensus.cc:482] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:48:17.077904 13608 raft_consensus.cc:504] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:17.078434 13608 leader_election.cc:288] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461), 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189)
W20260923 14:48:17.079282 13206 leader_election.cc:341] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461): Not found: Tablet not found: 3e69a97ce2ae4c5cac259582d28d229d
W20260923 14:48:17.079387 13204 leader_election.cc:341] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189): Not found: Tablet not found: 3e69a97ce2ae4c5cac259582d28d229d
I20260923 14:48:17.079443 13204 leader_election.cc:302] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7889a3e96284cfba0427850c5e08124; no voters: 5e1fd031be08463cbda4dfc74eaef121, abefccf81418440b8b4bc6edb53e0c14
I20260923 14:48:17.079625 13608 raft_consensus.cc:2726] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:48:17.280591 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:17.280169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:33453)
0923 14:48:17.280418 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:17.280490 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260923 14:48:17.452558 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:17.452110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:38889)
0923 14:48:17.452398 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:17.452459 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":72,"threads_started":1}
I20260923 14:48:17.641536 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:17.641100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:48671)
0923 14:48:17.641383 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:17.641444 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260923 14:48:18.281857 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:18.281430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:57263)
0923 14:48:18.281707 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:18.281759 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
I20260923 14:48:18.453754 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:18.453289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:42387)
0923 14:48:18.453600 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:18.453658 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":72,"threads_started":1}
I20260923 14:48:18.642721 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:18.642278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:43737)
0923 14:48:18.642569 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:18.642633 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":69,"threads_started":1}
I20260923 14:48:19.283113 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:19.282678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:34713)
0923 14:48:19.282958 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:19.283014 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
I20260923 14:48:19.454962 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:19.454555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:41819)
0923 14:48:19.454811 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:19.454865 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":94,"threads_started":1}
I20260923 14:48:19.644110 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:19.643553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:58487)
0923 14:48:19.643948 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:19.644009 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":99,"threads_started":1}
I20260923 14:48:20.284443 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:20.283948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:41317)
0923 14:48:20.284263 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:20.284319 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":91,"threads_started":1}
I20260923 14:48:20.456465 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:20.455735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:40975)
0923 14:48:20.456269 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:20.456346 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":167,"threads_started":1}
I20260923 14:48:20.645267 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:20.644835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:46307)
0923 14:48:20.645120 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:20.645172 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260923 14:48:21.285765 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:21.285319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:56363)
0923 14:48:21.285611 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:21.285666 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":97,"threads_started":1}
I20260923 14:48:21.367723 13622 raft_consensus.cc:482] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:48:21.367985 13622 raft_consensus.cc:504] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:21.368489 13622 leader_election.cc:288] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461), 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189)
W20260923 14:48:21.369233 13204 leader_election.cc:341] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189): Not found: Tablet not found: 3e69a97ce2ae4c5cac259582d28d229d
W20260923 14:48:21.369318 13206 leader_election.cc:341] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461): Not found: Tablet not found: 3e69a97ce2ae4c5cac259582d28d229d
I20260923 14:48:21.369433 13206 leader_election.cc:302] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7889a3e96284cfba0427850c5e08124; no voters: 5e1fd031be08463cbda4dfc74eaef121, abefccf81418440b8b4bc6edb53e0c14
I20260923 14:48:21.369612 13622 raft_consensus.cc:2726] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:48:21.457716 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:21.457269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:54225)
0923 14:48:21.457556 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:21.457615 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":89,"threads_started":1}
I20260923 14:48:21.646562 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:21.646069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:32831)
0923 14:48:21.646391 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:21.646460 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":85,"threads_started":1}
I20260923 14:48:22.287302 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:22.286758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:44191)
0923 14:48:22.287056 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:22.287196 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260923 14:48:22.458964 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:22.458548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:51623)
0923 14:48:22.458816 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:22.458869 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260923 14:48:22.647742 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:22.647324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:48321)
0923 14:48:22.647591 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:22.647646 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260923 14:48:23.288486 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:23.288059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:42029)
0923 14:48:23.288323 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:23.288381 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260923 14:48:23.460278 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:23.459772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:57995)
0923 14:48:23.460125 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:23.460180 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":109,"threads_started":1}
I20260923 14:48:23.649176 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:23.648681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:36815)
0923 14:48:23.648993 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:23.649072 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260923 14:48:23.743750 13631 raft_consensus.cc:482] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:48:23.744122 13631 raft_consensus.cc:504] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abefccf81418440b8b4bc6edb53e0c14" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 41461 } } peers { permanent_uuid: "5e1fd031be08463cbda4dfc74eaef121" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 45189 } } peers { permanent_uuid: "c7889a3e96284cfba0427850c5e08124" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 42783 } }
I20260923 14:48:23.744620 13631 leader_election.cc:288] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461), 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189)
W20260923 14:48:23.745421 13206 leader_election.cc:341] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abefccf81418440b8b4bc6edb53e0c14 (127.30.2.195:41461): Not found: Tablet not found: 3e69a97ce2ae4c5cac259582d28d229d
W20260923 14:48:23.745539 13204 leader_election.cc:341] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1fd031be08463cbda4dfc74eaef121 (127.30.2.194:45189): Not found: Tablet not found: 3e69a97ce2ae4c5cac259582d28d229d
I20260923 14:48:23.745651 13204 leader_election.cc:302] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7889a3e96284cfba0427850c5e08124; no voters: 5e1fd031be08463cbda4dfc74eaef121, abefccf81418440b8b4bc6edb53e0c14
I20260923 14:48:23.745841 13631 raft_consensus.cc:2726] T 3e69a97ce2ae4c5cac259582d28d229d P c7889a3e96284cfba0427850c5e08124 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:48:24.289628 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:24.289214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:35871)
0923 14:48:24.289469 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:24.289526 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260923 14:48:24.461521 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:24.461095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:44437)
0923 14:48:24.461358 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:24.461416 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260923 14:48:24.650332 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:24.649925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:54733)
0923 14:48:24.650185 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:24.650238 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260923 14:48:25.290879 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:25.290405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:44761)
0923 14:48:25.290690 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:25.290746 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":110,"threads_started":1}
I20260923 14:48:25.462692 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:25.462288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:38361)
0923 14:48:25.462543 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:25.462597 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":90,"threads_started":1}
I20260923 14:48:25.651499 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:25.651072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:56369)
0923 14:48:25.651332 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:25.651389 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":68,"threads_started":1}
I20260923 14:48:26.292337 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:26.291891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.193:42015)
0923 14:48:26.292193 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:26.292243 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20260923 14:48:26.464339 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:26.463896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.194:41247)
0923 14:48:26.464183 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:26.464242 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":110,"threads_started":1}
I20260923 14:48:26.638013 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 13154
I20260923 14:48:26.644711 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 13321
I20260923 14:48:26.652557 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 13455
I20260923 14:48:26.652944 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:26.652477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33037 (local address 127.30.2.195:36399)
0923 14:48:26.652745 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:26.652808 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
2026-09-23T14:48:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19439 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:48:26 dist-test-slave-wjf2 krb5kdc[13643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:48:26 dist-test-slave-wjf2 krb5kdc[13643](info): set up 2 sockets
Sep 23 14:48:26 dist-test-slave-wjf2 krb5kdc[13643](info): commencing operation
krb5kdc: starting...
W20260923 14:48:27.709344 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:48:27 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:48:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:48:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:48:27.831673 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45543
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:45543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:27.940845 13656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:27.941108 13656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:27.941174 13656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:27.945150 13656 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 14:48:27.945196 13656 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 14:48:27.945212 13656 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 14:48:27.945226 13656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:48:27.945271 13656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:27.945305 13656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:27.945331 13656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:48:27.945370 13656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:48:27.950651 13656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:45543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:27.951704 13656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:27.952757 13656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:27.961408 13662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:27.961359 13661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:27.961386 13656 server_base.cc:1034] running on GCE node
W20260923 14:48:27.961344 13664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:27.962211 13656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:27.963255 13656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:27.964437 13656 hybrid_clock.cc:648] HybridClock initialized: now 1790174907964426 us; error 30 us; skew 500 ppm
Sep 23 14:48:27 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174907, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:27.967080 13656 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:48:27.967963 13656 webserver.cc:460] Webserver started at http://127.30.2.254:40745/ using document root <none> and password file <none>
I20260923 14:48:27.968515 13656 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:27.968600 13656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:27.968818 13656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:27.970777 13656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "c3f5ca6ab39c4fd88e0510234e5757bd"
format_stamp: "Formatted at 2026-09-23 14:48:27 on dist-test-slave-wjf2"
server_key: "cbd60ad78662dc59adbb5b3f6f188669"
server_key_iv: "e2dc38bd7e9f7584c78197a2aa6d632a"
server_key_version: "encryptionkey@0"
I20260923 14:48:27.971313 13656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "c3f5ca6ab39c4fd88e0510234e5757bd"
format_stamp: "Formatted at 2026-09-23 14:48:27 on dist-test-slave-wjf2"
server_key: "cbd60ad78662dc59adbb5b3f6f188669"
server_key_iv: "e2dc38bd7e9f7584c78197a2aa6d632a"
server_key_version: "encryptionkey@0"
I20260923 14:48:27.975026 13656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260923 14:48:27.977504 13671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:27.978713 13656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:48:27.978855 13656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "c3f5ca6ab39c4fd88e0510234e5757bd"
format_stamp: "Formatted at 2026-09-23 14:48:27 on dist-test-slave-wjf2"
server_key: "cbd60ad78662dc59adbb5b3f6f188669"
server_key_iv: "e2dc38bd7e9f7584c78197a2aa6d632a"
server_key_version: "encryptionkey@0"
I20260923 14:48:27.978978 13656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:28.005998 13656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:28.009222 13656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:28.009425 13656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:28.016573 13656 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:45543
I20260923 14:48:28.016857 13723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:45543 every 8 connection(s)
I20260923 14:48:28.017545 13656 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:48:28.020522 13724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:28.026257 13724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Bootstrap starting.
I20260923 14:48:28.027662 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 13656
I20260923 14:48:28.027835 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:48:28.028152 30731 external_mini_cluster.cc:1428] Setting key e1fc20fdac48f673879171154532ac43
I20260923 14:48:28.028934 13724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:28.029731 13724 log.cc:824] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:28.031914 13724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: No bootstrap required, opened a new log
I20260923 14:48:28.036089 13724 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:48:28.036391 13724 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:28.036466 13724 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f5ca6ab39c4fd88e0510234e5757bd, State: Initialized, Role: FOLLOWER
I20260923 14:48:28.036921 13724 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:48:28.037048 13724 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:48:28.037129 13724 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:48:28.037240 13724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 0 FOLLOWER]: Advancing to term 1
Sep 23 14:48:28 dist-test-slave-wjf2 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:28.038071 13724 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:48:28.038446 13724 leader_election.cc:302] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f5ca6ab39c4fd88e0510234e5757bd; no voters:
I20260923 14:48:28.038745 13724 leader_election.cc:288] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:48:28.038852 13729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:48:28.039121 13729 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 1 LEADER]: Becoming Leader. State: Replica: c3f5ca6ab39c4fd88e0510234e5757bd, State: Running, Role: LEADER
I20260923 14:48:28.039500 13729 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:48:28.040098 13724 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:48:28.040762 13731 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f5ca6ab39c4fd88e0510234e5757bd. Latest consensus state: current_term: 1 leader_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } } }
I20260923 14:48:28.040901 13731 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:28.041565 13730 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } } }
I20260923 14:48:28.041673 13730 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:28.043668 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.029298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.30.2.254:45543)
0923 14:48:28.029764 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:28.029780 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:28.029840 (+ 60us) server_negotiation.cc:407] Connection header received
0923 14:48:28.030648 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:28.030668 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:28.030991 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:28.031268 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:28.032200 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.033178 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.034375 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.034827 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.034948 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.037872 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:28.037883 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:28.037897 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:28.037923 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:28.040087 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:28.040623 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:28.040629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:28.040634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:28.040777 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:28.041131 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:28.041135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:28.041140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:28.041703 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:28.042528 (+ 825us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:28.042676 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 14:48:28.042955 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":157,"threads_started":1}
W20260923 14:48:28.045547 13744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:48:28.045621 13744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:48:28.045680 13738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:48:28.046540 13738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:48:28.051985 13738 catalog_manager.cc:1347] Generated new cluster ID: cff97b2b81ec42e2a8c059aadcda5c78
I20260923 14:48:28.052042 13738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:48:28.061874 13738 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:48:28.063001 13738 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:48:28.073078 13738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Generated new TSK 0
I20260923 14:48:28.073802 13738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:48:28.107527 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45543
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:28.214911 13750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:28.215139 13750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:28.215179 13750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:28.219113 13750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:28.219161 13750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:28.219247 13750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:48:28.224365 13750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:28.225320 13750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:28.226305 13750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:28.235973 13755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:28.235978 13758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:28.236027 13750 server_base.cc:1034] running on GCE node
W20260923 14:48:28.235987 13756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:28.236522 13750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:28.237121 13750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:28.238313 13750 hybrid_clock.cc:648] HybridClock initialized: now 1790174908238296 us; error 26 us; skew 500 ppm
Sep 23 14:48:28 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174908, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:28.241058 13750 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:48:28.242033 13750 webserver.cc:460] Webserver started at http://127.30.2.193:43957/ using document root <none> and password file <none>
I20260923 14:48:28.242615 13750 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:28.242697 13750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:28.242895 13750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:28.244946 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "9b34f2c2a8a048198ac2aa410f1c6739"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "5ae5461856cbd1188d416b80ed8e033e"
server_key_iv: "06c49f1591d2224c58096ac9ccd6dbe8"
server_key_version: "encryptionkey@0"
I20260923 14:48:28.245501 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "9b34f2c2a8a048198ac2aa410f1c6739"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "5ae5461856cbd1188d416b80ed8e033e"
server_key_iv: "06c49f1591d2224c58096ac9ccd6dbe8"
server_key_version: "encryptionkey@0"
I20260923 14:48:28.249146 13750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:48:28.251423 13765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:28.252636 13750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:48:28.252767 13750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "9b34f2c2a8a048198ac2aa410f1c6739"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "5ae5461856cbd1188d416b80ed8e033e"
server_key_iv: "06c49f1591d2224c58096ac9ccd6dbe8"
server_key_version: "encryptionkey@0"
I20260923 14:48:28.252872 13750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:28.271217 13750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:28.274209 13750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:28.274394 13750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:28.274989 13750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:28.275771 13750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:28.275884 13750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:28.275987 13750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:28.276026 13750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:28.285833 13750 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:46091
I20260923 14:48:28.285851 13878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:46091 every 8 connection(s)
I20260923 14:48:28.286721 13750 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:48:28.293601 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 13750
I20260923 14:48:28.293728 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:48:28.294075 30731 external_mini_cluster.cc:1428] Setting key 70cf6c327ce1fb32a76b41aac7a42914
Sep 23 14:48:28 dist-test-slave-wjf2 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174908, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:28.300165 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.288527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:42287 (local address 127.30.2.254:45543)
0923 14:48:28.288656 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:28.288659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:28.289606 (+ 947us) server_negotiation.cc:407] Connection header received
0923 14:48:28.290407 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:28.290409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:28.290457 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:28.290528 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:28.291723 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.292326 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.293427 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.293798 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.293929 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.296272 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:28.296279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:28.296284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:28.296316 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:28.297747 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:28.298373 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:28.298379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:28.298383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:28.298472 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:28.298883 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:28.298889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:28.298894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:28.299146 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:28.299249 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:28.299773 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 14:48:28.299936 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:48:28.300932 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.288839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:42287)
0923 14:48:28.289485 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:28.289524 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:28.290234 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:28.290767 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:28.290774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:28.291124 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:28.291588 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.291602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.292441 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.292443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.293314 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.293324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.293949 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.293952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.294092 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.294695 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:28.294710 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:28.296119 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:28.297856 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:28.297864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:28.297880 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:28.298267 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:28.298642 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:28.298644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:28.298646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:28.298783 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:28.299256 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:28.299261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:28.299488 (+ 227us) client_negotiation.cc:769] Sending connection context
0923 14:48:28.299699 (+ 211us) client_negotiation.cc:240] Negotiation successful
0923 14:48:28.300025 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":121,"threads_started":1}
I20260923 14:48:28.302155 13879 heartbeater.cc:347] Connected to a master server at 127.30.2.254:45543
I20260923 14:48:28.302403 13879 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:28.302867 13879 heartbeater.cc:511] Master 127.30.2.254:45543 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:48:28.304382 13688 ts_manager.cc:194] Registered new tserver with Master: 9b34f2c2a8a048198ac2aa410f1c6739 (127.30.2.193:46091)
I20260923 14:48:28.305481 13688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:42287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:48:28.320796 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45543
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:28.433511 13884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:28.433753 13884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:28.433825 13884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:28.438484 13884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:28.438539 13884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:28.438632 13884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:48:28.444084 13884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:28.445096 13884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:28.446132 13884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:28.455552 13889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:28.455565 13892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:28.455682 13884 server_base.cc:1034] running on GCE node
W20260923 14:48:28.455633 13890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:28.456204 13884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:28.456880 13884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:28.458063 13884 hybrid_clock.cc:648] HybridClock initialized: now 1790174908458047 us; error 27 us; skew 500 ppm
Sep 23 14:48:28 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174908, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:28.460750 13884 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:48:28.461603 13884 webserver.cc:460] Webserver started at http://127.30.2.194:39251/ using document root <none> and password file <none>
I20260923 14:48:28.462153 13884 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:28.462239 13884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:28.462467 13884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:28.464511 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "e6ee306f97d3429c913c520e0c702648"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "6ea1fdf1cb642bdf20d54ba1b9f4b18f"
server_key_iv: "c3d0384e97f44dfafaddff263dedcd42"
server_key_version: "encryptionkey@0"
I20260923 14:48:28.465060 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "e6ee306f97d3429c913c520e0c702648"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "6ea1fdf1cb642bdf20d54ba1b9f4b18f"
server_key_iv: "c3d0384e97f44dfafaddff263dedcd42"
server_key_version: "encryptionkey@0"
I20260923 14:48:28.468878 13884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:48:28.471203 13899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:28.472512 13884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:48:28.472637 13884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "e6ee306f97d3429c913c520e0c702648"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "6ea1fdf1cb642bdf20d54ba1b9f4b18f"
server_key_iv: "c3d0384e97f44dfafaddff263dedcd42"
server_key_version: "encryptionkey@0"
I20260923 14:48:28.472741 13884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:28.496933 13884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:28.499929 13884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:28.500129 13884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:28.500744 13884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:28.501608 13884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:28.501696 13884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:28.501802 13884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:28.501850 13884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:28.510926 13884 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:44319
I20260923 14:48:28.510962 14012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:44319 every 8 connection(s)
I20260923 14:48:28.511848 13884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:48:28.516393 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 13884
I20260923 14:48:28.516547 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:48:28.516875 30731 external_mini_cluster.cc:1428] Setting key 448bd7dbe14e01f50aff618b93de9ba5
Sep 23 14:48:28 dist-test-slave-wjf2 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174908, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:48:28.527434 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.513823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:60897 (local address 127.30.2.254:45543)
0923 14:48:28.513965 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:28.513969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:28.515232 (+ 1263us) server_negotiation.cc:407] Connection header received
0923 14:48:28.516290 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:28.516294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:28.516349 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:28.516457 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:28.517966 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.518594 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.520428 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.520805 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.520924 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.523734 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:28.523738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:28.523740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:28.523767 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:28.525240 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:28.525861 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:28.525863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:28.525864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:28.525935 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:28.526321 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:28.526324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:28.526325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:28.526495 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:28.526567 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:28.527214 (+ 647us) server_negotiation.cc:299] Negotiation successful
0923 14:48:28.527310 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 14:48:28.528381 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.514219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:60897)
0923 14:48:28.515093 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:28.515135 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:28.516011 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:28.516679 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:28.516691 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:28.517206 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:28.517789 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.517809 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.518723 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.518728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.520282 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.520296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.520947 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.520974 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.521120 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.521708 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:28.521724 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:28.523545 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:28.525360 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:28.525369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:28.525387 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:28.525763 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:28.526027 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:28.526031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:28.526033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:28.526224 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:28.526606 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:28.526623 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:28.526941 (+ 318us) client_negotiation.cc:769] Sending connection context
0923 14:48:28.527180 (+ 239us) client_negotiation.cc:240] Negotiation successful
0923 14:48:28.527472 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":548,"threads_started":1}
I20260923 14:48:28.529536 14013 heartbeater.cc:347] Connected to a master server at 127.30.2.254:45543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
I20260923 14:48:28.529805 14013 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:28.530287 14013 heartbeater.cc:511] Master 127.30.2.254:45543 requested a full tablet report, sending...
I20260923 14:48:28.531424 13688 ts_manager.cc:194] Registered new tserver with Master: e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319)
I20260923 14:48:28.532075 13688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:60897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:48:28.547264 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45543
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:28.667222 14018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:28.667472 14018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:28.667520 14018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:28.671535 14018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:28.671599 14018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:28.671694 14018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:48:28.677060 14018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:45543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:28.678128 14018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:28.679188 14018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:28.689217 14024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:28.689258 14018 server_base.cc:1034] running on GCE node
W20260923 14:48:28.689213 14023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:28.689277 14026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:28.689899 14018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:28.690562 14018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:28.691745 14018 hybrid_clock.cc:648] HybridClock initialized: now 1790174908691727 us; error 28 us; skew 500 ppm
Sep 23 14:48:28 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174908, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:28.694459 14018 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:48:28.695272 14018 webserver.cc:460] Webserver started at http://127.30.2.195:37651/ using document root <none> and password file <none>
I20260923 14:48:28.695847 14018 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:28.695910 14018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:28.696098 14018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:48:28.698053 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "e036e2b041ae43889e4134d09eae6528"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "74d8779d4a8d76702656be0e3644b0b0"
server_key_iv: "8eda4e5a6af93f5c648e3ecafd2a5794"
server_key_version: "encryptionkey@0"
I20260923 14:48:28.698556 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "e036e2b041ae43889e4134d09eae6528"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "74d8779d4a8d76702656be0e3644b0b0"
server_key_iv: "8eda4e5a6af93f5c648e3ecafd2a5794"
server_key_version: "encryptionkey@0"
I20260923 14:48:28.702170 14018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260923 14:48:28.704491 14033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:28.705693 14018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:48:28.705785 14018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "e036e2b041ae43889e4134d09eae6528"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "74d8779d4a8d76702656be0e3644b0b0"
server_key_iv: "8eda4e5a6af93f5c648e3ecafd2a5794"
server_key_version: "encryptionkey@0"
I20260923 14:48:28.705865 14018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:28.727039 14018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:28.729882 14018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:28.730050 14018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:28.730594 14018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:28.731400 14018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:28.731448 14018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:28.731488 14018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:28.731503 14018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:28.740154 14018 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:43731
I20260923 14:48:28.740178 14146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:43731 every 8 connection(s)
I20260923 14:48:28.741181 14018 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:48:28.743901 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 14018
I20260923 14:48:28.744045 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:48:28.744361 30731 external_mini_cluster.cc:1428] Setting key 5ef25db760a75c5a0c7c94241c6e9a9a
Sep 23 14:48:28 dist-test-slave-wjf2 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174908, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:28.754161 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.742980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:53803 (local address 127.30.2.254:45543)
0923 14:48:28.743125 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:28.743129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:28.744283 (+ 1154us) server_negotiation.cc:407] Connection header received
0923 14:48:28.745373 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:28.745376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:28.745434 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:28.745533 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:28.746819 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.747346 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.748525 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.748736 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.748813 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.751143 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:28.751146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:28.751148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:28.751172 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:28.752613 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:28.753041 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:28.753044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:28.753045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:28.753109 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:28.753358 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:28.753360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:28.753361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:28.753518 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:28.753585 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:28.753969 (+ 384us) server_negotiation.cc:299] Negotiation successful
0923 14:48:28.754051 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:48:28.754935 14149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.743329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:53803)
0923 14:48:28.744124 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:28.744182 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:28.745143 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:28.745713 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:28.745722 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:28.746079 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:28.746694 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.746709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.747477 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.747480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.748417 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.748429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.748859 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.748862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.749017 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.749518 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:28.749534 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:28.750987 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:28.752711 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:28.752716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:28.752727 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:28.752969 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:28.753176 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:28.753178 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:28.753180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:28.753292 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:28.753594 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:28.753598 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:28.753791 (+ 193us) client_negotiation.cc:769] Sending connection context
0923 14:48:28.753954 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 14:48:28.754170 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":124,"threads_started":1}
I20260923 14:48:28.756126 14147 heartbeater.cc:347] Connected to a master server at 127.30.2.254:45543
I20260923 14:48:28.756362 14147 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:28.756934 14147 heartbeater.cc:511] Master 127.30.2.254:45543 requested a full tablet report, sending...
I20260923 14:48:28.758152 13687 ts_manager.cc:194] Registered new tserver with Master: e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731)
I20260923 14:48:28.758738 13687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:53803
I20260923 14:48:28.769901 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:48:28.779317 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.771849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.30.2.254:45543)
0923 14:48:28.771983 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:28.771989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:28.772102 (+ 113us) server_negotiation.cc:407] Connection header received
0923 14:48:28.772219 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:28.772221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:28.772261 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:28.772324 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:28.772939 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.773480 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.774519 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.774725 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.774798 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.776357 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:28.776360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:28.776362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:28.776385 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:28.777767 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:28.778153 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:28.778155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:28.778157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:28.778225 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:28.778464 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:28.778466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:28.778467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:28.778663 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:28.778761 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:28.779019 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 14:48:28.779107 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:48:28.782207 13687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:48:28.785281 13687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:48:28.800352 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.793816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:43731 (local address 127.0.0.1:47358)
0923 14:48:28.794576 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:28.794631 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:28.794764 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:28.796010 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:28.796019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:28.796042 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:28.796181 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.796187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.797720 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.797723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.798656 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.798666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.799739 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.799743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.799905 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.799953 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 14:48:28.800074 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 14:48:28.800200 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"spinlock_wait_cycles":6016,"thread_start_us":115,"threads_started":1}
I20260923 14:48:28.800467 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.795026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.30.2.195:43731)
0923 14:48:28.795521 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:28.795533 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:28.795555 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:48:28.795628 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:28.795632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:28.795840 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:28.796356 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:28.796518 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.797586 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.798781 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.799610 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.799723 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.799912 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:28.800208 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 14:48:28.800320 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":97,"threads_started":1}
I20260923 14:48:28.800545 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.793770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:44319 (local address 127.0.0.1:40060)
0923 14:48:28.794576 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:28.794631 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:28.794764 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:28.795541 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:28.795548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:28.795609 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:28.795827 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.795862 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.797390 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.797396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.798793 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.798805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.800117 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.800121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.800241 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.800264 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 14:48:28.800334 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 14:48:28.800400 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":22,"spinlock_wait_cycles":3840,"thread_start_us":68,"threads_started":1}
I20260923 14:48:28.801007 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.794495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:46091 (local address 127.0.0.1:42446)
0923 14:48:28.795199 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:28.795212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:28.795292 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:28.796071 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:28.796073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:28.796094 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:28.796294 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.796329 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.798134 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.798137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.799151 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.799158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.800117 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.800121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.800698 (+ 577us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.800728 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 14:48:28.800809 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 14:48:28.800876 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":472,"spinlock_wait_cycles":1280,"thread_start_us":94,"threads_started":1}
I20260923 14:48:28.801069 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.793895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.30.2.194:44319)
0923 14:48:28.794347 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:28.794361 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:28.794903 (+ 542us) server_negotiation.cc:407] Connection header received
0923 14:48:28.794995 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:28.795004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:28.795239 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:28.795404 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:28.796514 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.797258 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.798926 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.799731 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.799905 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.800073 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:28.800837 (+ 764us) server_negotiation.cc:299] Negotiation successful
0923 14:48:28.800932 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":120,"threads_started":1}
I20260923 14:48:28.801131 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.794671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.30.2.193:46091)
0923 14:48:28.795538 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:28.795556 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:28.795580 (+ 24us) server_negotiation.cc:407] Connection header received
0923 14:48:28.795663 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:28.795672 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:28.795926 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:28.796094 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:28.796501 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.797983 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.799263 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.799944 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.800099 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.800238 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:28.800866 (+ 628us) server_negotiation.cc:299] Negotiation successful
0923 14:48:28.800967 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":114,"threads_started":1}
I20260923 14:48:28.803642 14081 tablet_service.cc:1467] Processing CreateTablet for tablet 69462f7d85624e0ca8f8e63f2a9e151f (DEFAULT_TABLE table=test-table [id=3fa55698cd0e44cb9da32dd228a24342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:28.803646 13947 tablet_service.cc:1467] Processing CreateTablet for tablet 69462f7d85624e0ca8f8e63f2a9e151f (DEFAULT_TABLE table=test-table [id=3fa55698cd0e44cb9da32dd228a24342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:28.804669 14081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69462f7d85624e0ca8f8e63f2a9e151f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:28.804710 13947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69462f7d85624e0ca8f8e63f2a9e151f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:28.805619 13811 tablet_service.cc:1467] Processing CreateTablet for tablet 69462f7d85624e0ca8f8e63f2a9e151f (DEFAULT_TABLE table=test-table [id=3fa55698cd0e44cb9da32dd228a24342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:28.806532 13811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69462f7d85624e0ca8f8e63f2a9e151f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:28.810864 14164 tablet_bootstrap.cc:492] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: Bootstrap starting.
I20260923 14:48:28.812235 14165 tablet_bootstrap.cc:492] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: Bootstrap starting.
I20260923 14:48:28.813616 14166 tablet_bootstrap.cc:492] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Bootstrap starting.
I20260923 14:48:28.813670 14164 tablet_bootstrap.cc:654] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:28.814761 14165 tablet_bootstrap.cc:654] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:28.815073 14164 log.cc:824] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:28.815295 14166 tablet_bootstrap.cc:654] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:28.815551 14165 log.cc:824] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:28.816187 14166 log.cc:824] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:28.817092 14164 tablet_bootstrap.cc:492] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: No bootstrap required, opened a new log
I20260923 14:48:28.817279 14164 ts_tablet_manager.cc:1397] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260923 14:48:28.817473 14165 tablet_bootstrap.cc:492] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: No bootstrap required, opened a new log
I20260923 14:48:28.817694 14165 ts_tablet_manager.cc:1397] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:48:28.818037 14166 tablet_bootstrap.cc:492] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: No bootstrap required, opened a new log
I20260923 14:48:28.818218 14166 ts_tablet_manager.cc:1397] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:48:28.820415 14164 raft_consensus.cc:348] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:28.820662 14164 raft_consensus.cc:374] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:28.820713 14164 raft_consensus.cc:729] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e036e2b041ae43889e4134d09eae6528, State: Initialized, Role: FOLLOWER
I20260923 14:48:28.821137 14164 consensus_queue.cc:260] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:28.821743 14147 heartbeater.cc:503] Master 127.30.2.254:45543 was elected leader, sending a full tablet report...
I20260923 14:48:28.821846 14165 raft_consensus.cc:348] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:28.822090 14165 raft_consensus.cc:374] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:28.822113 14164 ts_tablet_manager.cc:1428] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260923 14:48:28.822141 14165 raft_consensus.cc:729] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ee306f97d3429c913c520e0c702648, State: Initialized, Role: FOLLOWER
I20260923 14:48:28.822569 14165 consensus_queue.cc:260] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:28.822518 14166 raft_consensus.cc:348] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:28.822743 14166 raft_consensus.cc:374] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:28.822808 14166 raft_consensus.cc:729] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b34f2c2a8a048198ac2aa410f1c6739, State: Initialized, Role: FOLLOWER
I20260923 14:48:28.823231 14013 heartbeater.cc:503] Master 127.30.2.254:45543 was elected leader, sending a full tablet report...
I20260923 14:48:28.823258 14166 consensus_queue.cc:260] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:28.823530 14165 ts_tablet_manager.cc:1428] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:48:28.824189 14166 ts_tablet_manager.cc:1428] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 14:48:28.824568 13879 heartbeater.cc:503] Master 127.30.2.254:45543 was elected leader, sending a full tablet report...
I20260923 14:48:28.938987 14171 raft_consensus.cc:482] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:48:28.939224 14171 raft_consensus.cc:504] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:28.940481 14171 leader_election.cc:288] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731), 9b34f2c2a8a048198ac2aa410f1c6739 (127.30.2.193:46091)
I20260923 14:48:28.944275 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.940755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:43731 (local address 127.30.2.194:60855)
0923 14:48:28.940879 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:28.940895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:28.940998 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:28.941368 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:28.941371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:28.941407 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:28.941522 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.941527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.942401 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.942404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.943166 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.943175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.943839 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.943842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.943956 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.943982 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 14:48:28.944068 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 14:48:28.944142 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 14:48:28.944275 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.940844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:60855 (local address 127.30.2.195:43731)
0923 14:48:28.940978 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:28.940981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:28.941027 (+ 46us) server_negotiation.cc:407] Connection header received
0923 14:48:28.941097 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:28.941101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:28.941177 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:28.941287 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:28.941660 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.942254 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.943298 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.943701 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.943847 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.943917 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:28.944087 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 14:48:28.944156 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:48:28.944715 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.941235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:48391 (local address 127.30.2.193:46091)
0923 14:48:28.941397 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:28.941404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:28.941605 (+ 201us) server_negotiation.cc:407] Connection header received
0923 14:48:28.941728 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:28.941732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:28.941781 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:28.941863 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:28.942137 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.942675 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.943689 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.944153 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.944273 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.944320 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:28.944550 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 14:48:28.944613 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 14:48:28.944730 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.941048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:46091 (local address 127.30.2.194:48391)
0923 14:48:28.941515 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:28.941526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:28.941634 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:28.941875 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:28.941877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:28.941892 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:28.942024 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.942029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.942786 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.942790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.943567 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:28.943575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.944273 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.944277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:28.944384 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.944417 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 14:48:28.944529 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 14:48:28.944594 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":1,"thread_start_us":166,"threads_started":1}
I20260923 14:48:28.945094 14101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69462f7d85624e0ca8f8e63f2a9e151f" candidate_uuid: "e6ee306f97d3429c913c520e0c702648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e036e2b041ae43889e4134d09eae6528" is_pre_election: true
I20260923 14:48:28.945437 14101 raft_consensus.cc:2454] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ee306f97d3429c913c520e0c702648 in term 0.
I20260923 14:48:28.945480 13832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69462f7d85624e0ca8f8e63f2a9e151f" candidate_uuid: "e6ee306f97d3429c913c520e0c702648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" is_pre_election: true
I20260923 14:48:28.945758 13832 raft_consensus.cc:2454] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ee306f97d3429c913c520e0c702648 in term 0.
I20260923 14:48:28.945962 13902 leader_election.cc:302] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e036e2b041ae43889e4134d09eae6528, e6ee306f97d3429c913c520e0c702648; no voters:
I20260923 14:48:28.946269 14171 raft_consensus.cc:2781] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:48:28.946373 14171 raft_consensus.cc:482] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:48:28.946434 14171 raft_consensus.cc:3037] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:28.947450 14171 raft_consensus.cc:504] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:28.947815 14171 leader_election.cc:288] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [CANDIDATE]: Term 1 election: Requested vote from peers e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731), 9b34f2c2a8a048198ac2aa410f1c6739 (127.30.2.193:46091)
I20260923 14:48:28.948238 13832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69462f7d85624e0ca8f8e63f2a9e151f" candidate_uuid: "e6ee306f97d3429c913c520e0c702648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b34f2c2a8a048198ac2aa410f1c6739"
I20260923 14:48:28.948238 14101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69462f7d85624e0ca8f8e63f2a9e151f" candidate_uuid: "e6ee306f97d3429c913c520e0c702648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e036e2b041ae43889e4134d09eae6528"
I20260923 14:48:28.948364 13832 raft_consensus.cc:3037] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:28.948364 14101 raft_consensus.cc:3037] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:28.949656 14101 raft_consensus.cc:2454] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ee306f97d3429c913c520e0c702648 in term 1.
I20260923 14:48:28.949656 13832 raft_consensus.cc:2454] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ee306f97d3429c913c520e0c702648 in term 1.
I20260923 14:48:28.949946 13902 leader_election.cc:302] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e036e2b041ae43889e4134d09eae6528, e6ee306f97d3429c913c520e0c702648; no voters:
I20260923 14:48:28.950105 14171 raft_consensus.cc:2781] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:48:28.950335 14171 raft_consensus.cc:686] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 1 LEADER]: Becoming Leader. State: Replica: e6ee306f97d3429c913c520e0c702648, State: Running, Role: LEADER
I20260923 14:48:28.950652 14171 consensus_queue.cc:237] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:28.953749 13688 catalog_manager.cc:5495] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ee306f97d3429c913c520e0c702648 (127.30.2.194). New cstate: current_term: 1 leader_uuid: "e6ee306f97d3429c913c520e0c702648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } health_report { overall_health: UNKNOWN } } }
I20260923 14:48:28.990859 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:28.987319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.30.2.194:44319)
0923 14:48:28.987447 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:28.987465 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:28.987480 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:48:28.987604 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:28.987607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:28.987661 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:28.987771 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:48:28.988045 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.988548 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.989428 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:28.989645 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:28.989750 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:28.990053 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:48:28.990461 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:48:28.990586 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:28.990647 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 14:48:28.990702 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260923 14:48:28.992116 14148 tablet.cc:2369] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:48:28.997141 14101 raft_consensus.cc:1261] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528 [term 1 FOLLOWER]: Refusing update from remote peer e6ee306f97d3429c913c520e0c702648: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:28.997193 13832 raft_consensus.cc:1261] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Refusing update from remote peer e6ee306f97d3429c913c520e0c702648: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:28.997915 14174 consensus_queue.cc:1035] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:48:28.998127 14171 consensus_queue.cc:1035] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:48:29.014622 13688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:48:29.014813 13688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:48:29.025107 13688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Sending DeleteTablet for 3 replicas of tablet 69462f7d85624e0ca8f8e63f2a9e151f
I20260923 14:48:29.025961 13947 tablet_service.cc:1514] Processing DeleteTablet for tablet 69462f7d85624e0ca8f8e63f2a9e151f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:29 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:40060
I20260923 14:48:29.026170 13811 tablet_service.cc:1514] Processing DeleteTablet for tablet 69462f7d85624e0ca8f8e63f2a9e151f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:29 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42446
I20260923 14:48:29.026176 14081 tablet_service.cc:1514] Processing DeleteTablet for tablet 69462f7d85624e0ca8f8e63f2a9e151f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:29 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:47358
I20260923 14:48:29.026700 14188 tablet_replica.cc:330] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: stopping tablet replica
I20260923 14:48:29.026829 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 13750
I20260923 14:48:29.028754 14188 raft_consensus.cc:2229] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:48:29.029140 14188 raft_consensus.cc:2258] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:29.031095 14188 ts_tablet_manager.cc:1905] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:29.032835 14187 tablet_replica.cc:330] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: stopping tablet replica
I20260923 14:48:29.033208 14187 raft_consensus.cc:2229] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:48:29.033684 14187 raft_consensus.cc:2258] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:29.035101 14188 ts_tablet_manager.cc:1918] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:29.035218 14188 log.cc:1196] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/69462f7d85624e0ca8f8e63f2a9e151f
I20260923 14:48:29.035159 14187 ts_tablet_manager.cc:1905] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:29.035629 14188 ts_tablet_manager.cc:1939] T 69462f7d85624e0ca8f8e63f2a9e151f P e036e2b041ae43889e4134d09eae6528: Deleting consensus metadata
W20260923 14:48:29.036123 13674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:46091 (error 108)
I20260923 14:48:29.037048 13674 catalog_manager.cc:4841] TS e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731): tablet 69462f7d85624e0ca8f8e63f2a9e151f (table test-table [id=3fa55698cd0e44cb9da32dd228a24342]) successfully deleted
I20260923 14:48:29.037281 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 13884
I20260923 14:48:29.038172 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:29.037415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:46091 (local address 127.0.0.1:42460)
0923 14:48:29.037982 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:29.038068 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458}
W20260923 14:48:29.039134 13674 catalog_manager.cc:4584] TS 9b34f2c2a8a048198ac2aa410f1c6739 (127.30.2.193:46091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69462f7d85624e0ca8f8e63f2a9e151f: Network error: Client connection negotiation failed: client connection to 127.30.2.193:46091: connect: Connection refused (error 111)
I20260923 14:48:29.043475 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 14018
I20260923 14:48:29.043756 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:29.043451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:44319 (local address 127.0.0.1:40086)
0923 14:48:29.043609 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:29.043670 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.194:44319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 14:48:29.048990 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 13656
I20260923 14:48:49.055490 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45543
--webserver_interface=127.30.2.254
--webserver_port=40745
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:45543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:48:49.165431 14191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:49.165700 14191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:49.165738 14191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:49.169839 14191 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 14:48:49.169891 14191 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 14:48:49.169907 14191 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 14:48:49.169921 14191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:48:49.169939 14191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:49.169956 14191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:49.170012 14191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:48:49.170037 14191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:48:49.175360 14191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:45543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=40745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:49.176376 14191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:49.177421 14191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:49.190968 14198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:49.191112 14191 server_base.cc:1034] running on GCE node
W20260923 14:48:49.190995 14197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:49.190995 14200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:49.191918 14191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:49.193037 14191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:49.194232 14191 hybrid_clock.cc:648] HybridClock initialized: now 1790174929194217 us; error 26 us; skew 500 ppm
Sep 23 14:48:49 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174929, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:49.203447 14191 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:48:49.204461 14191 webserver.cc:460] Webserver started at http://127.30.2.254:40745/ using document root <none> and password file <none>
I20260923 14:48:49.205139 14191 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:49.205243 14191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:49.209167 14191 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:48:49.211513 14207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:49.212829 14191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:48:49.212983 14191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "c3f5ca6ab39c4fd88e0510234e5757bd"
format_stamp: "Formatted at 2026-09-23 14:48:27 on dist-test-slave-wjf2"
server_key: "cbd60ad78662dc59adbb5b3f6f188669"
server_key_iv: "e2dc38bd7e9f7584c78197a2aa6d632a"
server_key_version: "encryptionkey@0"
I20260923 14:48:49.213591 14191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:49.265668 14191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:49.269562 14191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:49.269781 14191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:49.277150 14191 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:45543
I20260923 14:48:49.277464 14259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:45543 every 8 connection(s)
I20260923 14:48:49.278311 14191 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:48:49.282725 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 14191
I20260923 14:48:49.283387 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:46091
--local_ip_for_outbound_sockets=127.30.2.193
--tserver_master_addrs=127.30.2.254:45543
--webserver_port=43957
--webserver_interface=127.30.2.193
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:48:49.284545 14260 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:48:49.287274 14260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Bootstrap starting.
I20260923 14:48:49.305531 14260 log.cc:824] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:49.314903 14260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:48:49.315356 14260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Bootstrap complete.
I20260923 14:48:49.319775 14260 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:48:49.320291 14260 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f5ca6ab39c4fd88e0510234e5757bd, State: Initialized, Role: FOLLOWER
I20260923 14:48:49.320819 14260 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:48:49.320988 14260 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:48:49.321089 14260 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:48:49.321238 14260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:48:49.323184 14260 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:48:49.323544 14260 leader_election.cc:302] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f5ca6ab39c4fd88e0510234e5757bd; no voters:
I20260923 14:48:49.323947 14260 leader_election.cc:288] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 14:48:49.324057 14264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 2 FOLLOWER]: Leader election won for term 2
I20260923 14:48:49.324286 14264 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 2 LEADER]: Becoming Leader. State: Replica: c3f5ca6ab39c4fd88e0510234e5757bd, State: Running, Role: LEADER
I20260923 14:48:49.324591 14264 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:48:49.325237 14260 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:48:49.325565 14264 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f5ca6ab39c4fd88e0510234e5757bd. Latest consensus state: current_term: 2 leader_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } } }
I20260923 14:48:49.325698 14264 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:49.326035 14265 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } } }
I20260923 14:48:49.326200 14265 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: This master's current role is: LEADER
I20260923 14:48:49.326110 14268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:48:49.331251 14268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:48:49.331969 14268 catalog_manager.cc:1259] Loaded cluster ID: cff97b2b81ec42e2a8c059aadcda5c78
I20260923 14:48:49.332028 14268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:48:49.334332 14268 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:48:49.334991 14268 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Loaded TSK: 0
I20260923 14:48:49.356897 14268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Generated new TSK 1
I20260923 14:48:49.357177 14268 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:48:49.404134 14262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:49.404356 14262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:49.404397 14262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:49.408397 14262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:49.408452 14262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:49.408535 14262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:48:49.413591 14262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:46091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=43957
--tserver_master_addrs=127.30.2.254:45543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:49.414574 14262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:49.415570 14262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:49.425096 14287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:49.425180 14262 server_base.cc:1034] running on GCE node
W20260923 14:48:49.425110 14288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:49.425304 14290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:49.425805 14262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:49.426880 14262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:49.428104 14262 hybrid_clock.cc:648] HybridClock initialized: now 1790174929428071 us; error 47 us; skew 500 ppm
Sep 23 14:48:49 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174929, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:49.431475 14262 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:48:49.432349 14262 webserver.cc:460] Webserver started at http://127.30.2.193:43957/ using document root <none> and password file <none>
I20260923 14:48:49.432936 14262 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:49.433045 14262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:49.436941 14262 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:48:49.439198 14297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:49.440627 14262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:48:49.440759 14262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "9b34f2c2a8a048198ac2aa410f1c6739"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "5ae5461856cbd1188d416b80ed8e033e"
server_key_iv: "06c49f1591d2224c58096ac9ccd6dbe8"
server_key_version: "encryptionkey@0"
I20260923 14:48:49.441303 14262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:49.455874 14262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:49.459039 14262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:49.459240 14262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:49.459880 14262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:49.461052 14304 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 14:48:49.465155 14262 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 14:48:49.465220 14262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260923 14:48:49.465268 14262 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 14:48:49.467053 14262 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 14:48:49.467110 14262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260923 14:48:49.467377 14304 tablet_bootstrap.cc:492] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Bootstrap starting.
I20260923 14:48:49.478114 14262 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:46091
I20260923 14:48:49.478353 14411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:46091 every 8 connection(s)
I20260923 14:48:49.479302 14262 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:48:49.481004 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 14262
I20260923 14:48:49.481535 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:44319
--local_ip_for_outbound_sockets=127.30.2.194
--tserver_master_addrs=127.30.2.254:45543
--webserver_port=39251
--webserver_interface=127.30.2.194
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:48:49.488844 14304 log.cc:824] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Log is configured to *not* fsync() on all Append() calls
Sep 23 14:48:49 dist-test-slave-wjf2 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174929, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:49.497226 14304 tablet_bootstrap.cc:492] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:48:49.497645 14304 tablet_bootstrap.cc:492] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Bootstrap complete.
I20260923 14:48:49.498688 14415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.481634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:60557)
0923 14:48:49.483027 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:49.483072 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:49.483960 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:49.485113 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:49.485122 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:49.485560 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:49.486034 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.486049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.487407 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.487411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.488396 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.488407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.489236 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.489244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.489448 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.490198 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:49.490225 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:49.492231 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:49.495043 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:49.495050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:49.495063 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:49.495434 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:49.495910 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:49.495913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:49.495915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:49.496105 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:49.496848 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:49.496857 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:49.497179 (+ 322us) client_negotiation.cc:769] Sending connection context
0923 14:48:49.497386 (+ 207us) client_negotiation.cc:240] Negotiation successful
0923 14:48:49.497741 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"thread_start_us":147,"threads_started":1}
I20260923 14:48:49.498832 14304 ts_tablet_manager.cc:1397] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.008s
I20260923 14:48:49.500479 14414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.481608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:60557 (local address 127.30.2.254:45543)
0923 14:48:49.484053 (+ 2445us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:49.484075 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:49.484121 (+ 46us) server_negotiation.cc:407] Connection header received
0923 14:48:49.484399 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:49.484411 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:49.484705 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:49.485050 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:49.486228 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.487223 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.488550 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.489079 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.489281 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.492472 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:49.492481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:49.492499 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:49.492541 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:49.494906 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.495583 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.495591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.495597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.495755 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.496243 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.496247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.496249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.496708 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:49.496856 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:49.499358 (+ 2502us) server_negotiation.cc:299] Negotiation successful
0923 14:48:49.499647 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2301,"thread_start_us":139,"threads_started":1}
I20260923 14:48:49.501639 14304 raft_consensus.cc:348] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:49.502117 14304 raft_consensus.cc:729] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b34f2c2a8a048198ac2aa410f1c6739, State: Initialized, Role: FOLLOWER
I20260923 14:48:49.502658 14304 consensus_queue.cc:260] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:49.502928 14412 heartbeater.cc:347] Connected to a master server at 127.30.2.254:45543
I20260923 14:48:49.503173 14412 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:49.503731 14412 heartbeater.cc:511] Master 127.30.2.254:45543 requested a full tablet report, sending...
I20260923 14:48:49.503739 14304 ts_tablet_manager.cc:1428] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260923 14:48:49.506312 14224 ts_manager.cc:194] Registered new tserver with Master: 9b34f2c2a8a048198ac2aa410f1c6739 (127.30.2.193:46091)
I20260923 14:48:49.510252 14224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:60557
I20260923 14:48:49.512802 14412 heartbeater.cc:503] Master 127.30.2.254:45543 was elected leader, sending a full tablet report...
Sep 23 14:48:49 dist-test-slave-wjf2 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174929, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.193@KRBTEST.COM
I20260923 14:48:49.520720 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.509634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:46091 (local address 127.0.0.1:39866)
0923 14:48:49.510429 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:49.510474 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:49.510635 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:49.511186 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:49.511192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:49.511630 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:49.511760 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.511768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.513123 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.513127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.513888 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.513899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.514471 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.514475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.514589 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.515172 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:49.515184 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:49.516766 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:49.518899 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:49.518908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:49.518913 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:49.519144 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:49.519462 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:49.519465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:49.519467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:49.519556 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:49.520070 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:49.520076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:49.520201 (+ 125us) client_negotiation.cc:769] Sending connection context
0923 14:48:49.520352 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 14:48:49.520545 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":118,"threads_started":1}
I20260923 14:48:49.520750 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.509684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.30.2.193:46091)
0923 14:48:49.510154 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:49.510166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:49.510565 (+ 399us) server_negotiation.cc:407] Connection header received
0923 14:48:49.510791 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:49.510796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:49.511020 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:49.511157 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:49.511927 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.512998 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.514014 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.514349 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.514467 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.516909 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:49.516920 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:49.516934 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:49.516965 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:49.518759 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.519264 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.519270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.519272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.519356 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.519648 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.519653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.519655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.519955 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:49.520115 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:49.520444 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 14:48:49.520605 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":87,"threads_started":1}
I20260923 14:48:49.521795 14346 tablet_service.cc:1514] Processing DeleteTablet for tablet 69462f7d85624e0ca8f8e63f2a9e151f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:29 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:39866
I20260923 14:48:49.522743 14421 tablet_replica.cc:330] stopping tablet replica
I20260923 14:48:49.523041 14421 raft_consensus.cc:2229] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:48:49.523291 14421 raft_consensus.cc:2258] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:49.524290 14421 ts_tablet_manager.cc:1905] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:49.527961 14421 ts_tablet_manager.cc:1918] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:49.528066 14421 log.cc:1196] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal/wals/69462f7d85624e0ca8f8e63f2a9e151f
I20260923 14:48:49.528342 14421 ts_tablet_manager.cc:1939] T 69462f7d85624e0ca8f8e63f2a9e151f P 9b34f2c2a8a048198ac2aa410f1c6739: Deleting consensus metadata
I20260923 14:48:49.530081 14210 catalog_manager.cc:4841] TS 9b34f2c2a8a048198ac2aa410f1c6739 (127.30.2.193:46091): tablet 69462f7d85624e0ca8f8e63f2a9e151f (table test-table [id=3fa55698cd0e44cb9da32dd228a24342]) successfully deleted
W20260923 14:48:49.604909 14416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:49.605119 14416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:49.605185 14416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:49.609093 14416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:49.609140 14416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:49.609227 14416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:48:49.614225 14416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:44319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=39251
--tserver_master_addrs=127.30.2.254:45543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:49.615181 14416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:49.616183 14416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:49.625712 14429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:49.625735 14427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:49.625746 14426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:49.626014 14416 server_base.cc:1034] running on GCE node
I20260923 14:48:49.626382 14416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:49.627050 14416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:49.628207 14416 hybrid_clock.cc:648] HybridClock initialized: now 1790174929628177 us; error 42 us; skew 500 ppm
Sep 23 14:48:49 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174929, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:49.630941 14416 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:48:49.631863 14416 webserver.cc:460] Webserver started at http://127.30.2.194:39251/ using document root <none> and password file <none>
I20260923 14:48:49.632402 14416 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:49.632505 14416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:49.636133 14416 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260923 14:48:49.638206 14436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:49.639326 14416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:48:49.639444 14416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "e6ee306f97d3429c913c520e0c702648"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "6ea1fdf1cb642bdf20d54ba1b9f4b18f"
server_key_iv: "c3d0384e97f44dfafaddff263dedcd42"
server_key_version: "encryptionkey@0"
I20260923 14:48:49.639986 14416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:49.664090 14416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:49.667696 14416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:49.667938 14416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:49.668567 14416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:49.669677 14443 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260923 14:48:49.673751 14443 ts_tablet_manager.cc:1852] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260923 14:48:49.674190 14443 ts_tablet_manager.cc:1905] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:49.678315 14443 ts_tablet_manager.cc:1918] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:49.678407 14443 log.cc:1196] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/69462f7d85624e0ca8f8e63f2a9e151f
I20260923 14:48:49.678777 14443 ts_tablet_manager.cc:1939] T 69462f7d85624e0ca8f8e63f2a9e151f P e6ee306f97d3429c913c520e0c702648: Deleting consensus metadata
I20260923 14:48:49.679104 14416 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260923 14:48:49.679176 14416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260923 14:48:49.679277 14416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:49.679327 14416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:49.688318 14416 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:44319
I20260923 14:48:49.688365 14550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:44319 every 8 connection(s)
I20260923 14:48:49.689153 14416 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:48:49.692612 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 14416
I20260923 14:48:49.693073 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:43731
--local_ip_for_outbound_sockets=127.30.2.195
--tserver_master_addrs=127.30.2.254:45543
--webserver_port=37651
--webserver_interface=127.30.2.195
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:48:49 dist-test-slave-wjf2 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174929, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:49.704411 14414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.691098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:52087 (local address 127.30.2.254:45543)
0923 14:48:49.691284 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:49.691289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:49.692221 (+ 932us) server_negotiation.cc:407] Connection header received
0923 14:48:49.693243 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:49.693247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:49.693343 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:49.693474 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:49.694593 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.695468 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.696757 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.697228 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.697361 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.700195 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:49.700203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:49.700209 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:49.700246 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:49.702050 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.702583 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.702587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.702590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.702691 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.703053 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.703059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.703063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.703346 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:49.703456 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:49.704011 (+ 555us) server_negotiation.cc:299] Negotiation successful
0923 14:48:49.704173 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 14:48:49.705049 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.691427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:52087)
0923 14:48:49.692063 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:49.692114 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:49.693001 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:49.693617 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:49.693624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:49.693965 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:49.694440 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.694454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.695650 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.695653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.696625 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.696636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.697380 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.697383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.697533 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.698217 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:49.698233 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:49.700011 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:49.702185 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:49.702191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:49.702203 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:49.702456 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:49.702792 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:49.702795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:49.702796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:49.702930 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:49.703458 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:49.703464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:49.703694 (+ 230us) client_negotiation.cc:769] Sending connection context
0923 14:48:49.703895 (+ 201us) client_negotiation.cc:240] Negotiation successful
0923 14:48:49.704171 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
I20260923 14:48:49.706283 14551 heartbeater.cc:347] Connected to a master server at 127.30.2.254:45543
I20260923 14:48:49.706548 14551 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:49.707125 14551 heartbeater.cc:511] Master 127.30.2.254:45543 requested a full tablet report, sending...
I20260923 14:48:49.708305 14224 ts_manager.cc:194] Registered new tserver with Master: e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319)
I20260923 14:48:49.708879 14224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:52087
W20260923 14:48:49.806972 14554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:48:49.807236 14554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:48:49.807346 14554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:48:49.811383 14554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:48:49.811445 14554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:48:49.811549 14554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:48:49.816864 14554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:43731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=37651
--tserver_master_addrs=127.30.2.254:45543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:48:49.817981 14554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:48:49.819054 14554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:48:49.829012 14559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:49.829087 14560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:48:49.828989 14562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:48:49.829149 14554 server_base.cc:1034] running on GCE node
I20260923 14:48:49.829605 14554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:48:49.830263 14554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:48:49.831483 14554 hybrid_clock.cc:648] HybridClock initialized: now 1790174929831462 us; error 28 us; skew 500 ppm
Sep 23 14:48:49 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174929, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:48:49.834265 14554 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:48:49.835116 14554 webserver.cc:460] Webserver started at http://127.30.2.195:37651/ using document root <none> and password file <none>
I20260923 14:48:49.835681 14554 fs_manager.cc:359] Metadata directory not provided
I20260923 14:48:49.835783 14554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:48:49.839589 14554 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260923 14:48:49.841614 14569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:49.842871 14554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:48:49.843003 14554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "e036e2b041ae43889e4134d09eae6528"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "74d8779d4a8d76702656be0e3644b0b0"
server_key_iv: "8eda4e5a6af93f5c648e3ecafd2a5794"
server_key_version: "encryptionkey@0"
I20260923 14:48:49.843564 14554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:48:49.873122 14554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:48:49.876286 14554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:48:49.876498 14554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:48:49.877102 14554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:48:49.877930 14554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:48:49.877997 14554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:49.878065 14554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:48:49.878113 14554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:48:49.887538 14554 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:43731
I20260923 14:48:49.887574 14682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:43731 every 8 connection(s)
I20260923 14:48:49.888569 14554 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:48:49.890090 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 14554
Sep 23 14:48:49 dist-test-slave-wjf2 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174929, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:48:49.899613 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.890789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.30.2.254:45543)
0923 14:48:49.891074 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:49.891079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:49.891247 (+ 168us) server_negotiation.cc:407] Connection header received
0923 14:48:49.891439 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:49.891442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:49.891503 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:49.891598 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:49.892506 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.893193 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.894099 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.894330 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.894419 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.895948 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:49.895952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:49.895954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:49.895976 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:49.897606 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.898108 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.898111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.898112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.898198 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.898499 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.898502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.898503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.898755 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:49.898956 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:49.899239 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 14:48:49.899415 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"spinlock_wait_cycles":6912,"thread_start_us":94,"threads_started":1}
I20260923 14:48:49.902909 14414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.890448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:54485 (local address 127.30.2.254:45543)
0923 14:48:49.890622 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:49.890625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:49.891937 (+ 1312us) server_negotiation.cc:407] Connection header received
0923 14:48:49.892810 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:49.892813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:49.892856 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:49.892936 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:49.894050 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.894653 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.896130 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.896362 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.896485 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.899193 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:49.899196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:49.899199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:49.899231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:49.900974 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.901588 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.901591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.901593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.901702 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.902021 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.902031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.902032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.902184 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:49.902261 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:49.902703 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 14:48:49.902790 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53,"spinlock_wait_cycles":896}
I20260923 14:48:49.903862 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.890995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:54485)
0923 14:48:49.891753 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:49.891833 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:49.892631 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:49.893106 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:49.893112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:49.893446 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:48:49.893907 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.893919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.894816 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.894821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.895997 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.896008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.896523 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.896527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.896696 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.897313 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:48:49.897334 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:48:49.898986 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:48:49.901083 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:49.901089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:49.901103 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:49.901441 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:49.901798 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:48:49.901800 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:48:49.901801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:48:49.901938 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:48:49.902272 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:48:49.902277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:48:49.902498 (+ 221us) client_negotiation.cc:769] Sending connection context
0923 14:48:49.902651 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 14:48:49.902887 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":148,"threads_started":1}
I20260923 14:48:49.904888 14683 heartbeater.cc:347] Connected to a master server at 127.30.2.254:45543
I20260923 14:48:49.905098 14683 heartbeater.cc:464] Registering TS with master...
I20260923 14:48:49.905539 14683 heartbeater.cc:511] Master 127.30.2.254:45543 requested a full tablet report, sending...
I20260923 14:48:49.906505 14224 ts_manager.cc:194] Registered new tserver with Master: e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731)
I20260923 14:48:49.907070 14224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:54485
I20260923 14:48:49.913411 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:48:49.917757 14414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.914007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.30.2.254:45543)
0923 14:48:49.914118 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:49.914123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:49.914370 (+ 247us) server_negotiation.cc:407] Connection header received
0923 14:48:49.914464 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:49.914466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:49.914552 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:49.914629 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:48:49.915133 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.915605 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.916673 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.916888 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.916986 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.917309 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:48:49.917444 (+ 135us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 14:48:49.917659 (+ 215us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34958: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260923 14:48:49.917829 14414 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34958: token expired
W20260923 14:48:49.918223 14153 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.2.254:45543)
I20260923 14:48:49.925479 14414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.918600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.30.2.254:45543)
0923 14:48:49.918740 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:49.918743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:49.918756 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:48:49.918788 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:49.918790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:49.918858 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:49.918929 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:48:49.919505 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.920091 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.920852 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.921050 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.921134 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.922329 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:48:49.922333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:48:49.922335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:48:49.922357 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:48:49.924025 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.924382 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.924385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.924386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.924463 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:48:49.924690 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:48:49.924692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:48:49.924693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:48:49.924898 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:48:49.925008 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:49.925276 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 14:48:49.925378 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 14:48:49.927820 14224 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:48:49.931212 14224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:48:49.939853 14346 tablet_service.cc:1467] Processing CreateTablet for tablet eb3fce3c1aff4db09600aaa7560150a6 (DEFAULT_TABLE table=test-table [id=ef2d81a402e64f8cae4d0768effd252b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:49.940513 14346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3fce3c1aff4db09600aaa7560150a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:49.943143 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.937945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:43731 (local address 127.0.0.1:39970)
0923 14:48:49.938063 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:49.938078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:49.938150 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:49.939098 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:49.939101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:49.939122 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:49.939226 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.939232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.940801 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.940807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.941890 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.941906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.942715 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.942718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.942803 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.942842 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 14:48:49.942930 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 14:48:49.943004 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":5}
I20260923 14:48:49.943260 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.938125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.30.2.195:43731)
0923 14:48:49.938562 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:49.938581 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:49.938608 (+ 27us) server_negotiation.cc:407] Connection header received
0923 14:48:49.938693 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:49.938699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:49.938923 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:49.939381 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:49.939527 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.940695 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.942017 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.942593 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.942704 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.942845 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:49.942961 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 14:48:49.943100 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":178,"threads_started":1}
I20260923 14:48:49.944017 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.938532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:44319 (local address 127.0.0.1:49806)
0923 14:48:49.938817 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:49.938828 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:49.938917 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:49.939638 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:49.939641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:49.939657 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:49.939731 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.939736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.941057 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.941063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.942231 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:49.942243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.943606 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.943609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:49.943693 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.943714 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 14:48:49.943813 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 14:48:49.943889 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260923 14:48:49.944339 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:49.938831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.30.2.194:44319)
0923 14:48:49.939179 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:49.939192 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:49.939221 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:48:49.939284 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:49.939288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:49.939494 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:49.939642 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:49.939876 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.940919 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.942378 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:49.943257 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:49.943384 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:49.943542 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:49.944024 (+ 482us) server_negotiation.cc:299] Negotiation successful
0923 14:48:49.944169 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":123,"threads_started":1}
I20260923 14:48:49.945787 14485 tablet_service.cc:1467] Processing CreateTablet for tablet eb3fce3c1aff4db09600aaa7560150a6 (DEFAULT_TABLE table=test-table [id=ef2d81a402e64f8cae4d0768effd252b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:49.946339 14617 tablet_service.cc:1467] Processing CreateTablet for tablet eb3fce3c1aff4db09600aaa7560150a6 (DEFAULT_TABLE table=test-table [id=ef2d81a402e64f8cae4d0768effd252b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:48:49.946907 14485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3fce3c1aff4db09600aaa7560150a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:49.947386 14617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3fce3c1aff4db09600aaa7560150a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:48:49.948769 14304 tablet_bootstrap.cc:492] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: Bootstrap starting.
I20260923 14:48:49.949586 14304 tablet_bootstrap.cc:654] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:49.950846 14304 tablet_bootstrap.cc:492] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: No bootstrap required, opened a new log
I20260923 14:48:49.950954 14304 ts_tablet_manager.cc:1397] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 14:48:49.951355 14304 raft_consensus.cc:348] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:49.951483 14304 raft_consensus.cc:374] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:49.951530 14304 raft_consensus.cc:729] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b34f2c2a8a048198ac2aa410f1c6739, State: Initialized, Role: FOLLOWER
I20260923 14:48:49.951653 14304 consensus_queue.cc:260] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:49.951956 14304 ts_tablet_manager.cc:1428] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:48:49.951995 14443 tablet_bootstrap.cc:492] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648: Bootstrap starting.
I20260923 14:48:49.954118 14443 tablet_bootstrap.cc:654] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:49.954929 14443 log.cc:824] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:49.956637 14697 tablet_bootstrap.cc:492] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528: Bootstrap starting.
I20260923 14:48:49.956878 14443 tablet_bootstrap.cc:492] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648: No bootstrap required, opened a new log
I20260923 14:48:49.957068 14443 ts_tablet_manager.cc:1397] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:48:49.958436 14697 tablet_bootstrap.cc:654] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528: Neither blocks nor log segments found. Creating new log.
I20260923 14:48:49.959410 14697 log.cc:824] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528: Log is configured to *not* fsync() on all Append() calls
I20260923 14:48:49.960031 14443 raft_consensus.cc:348] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:49.960299 14443 raft_consensus.cc:374] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:49.960359 14443 raft_consensus.cc:729] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ee306f97d3429c913c520e0c702648, State: Initialized, Role: FOLLOWER
I20260923 14:48:49.960801 14443 consensus_queue.cc:260] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:49.961256 14697 tablet_bootstrap.cc:492] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528: No bootstrap required, opened a new log
I20260923 14:48:49.961419 14551 heartbeater.cc:503] Master 127.30.2.254:45543 was elected leader, sending a full tablet report...
I20260923 14:48:49.961468 14697 ts_tablet_manager.cc:1397] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:48:49.961736 14443 ts_tablet_manager.cc:1428] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:48:49.964905 14697 raft_consensus.cc:348] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:49.965096 14697 raft_consensus.cc:374] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:48:49.965175 14697 raft_consensus.cc:729] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e036e2b041ae43889e4134d09eae6528, State: Initialized, Role: FOLLOWER
I20260923 14:48:49.965551 14697 consensus_queue.cc:260] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:49.966198 14683 heartbeater.cc:503] Master 127.30.2.254:45543 was elected leader, sending a full tablet report...
I20260923 14:48:49.966563 14697 ts_tablet_manager.cc:1428] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260923 14:48:49.980985 14413 tablet.cc:2369] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:48:49.999521 14418 raft_consensus.cc:482] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:48:49.999794 14418 raft_consensus.cc:504] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:50.001094 14418 leader_election.cc:288] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731), e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319)
I20260923 14:48:50.005496 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:50.001495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:35121 (local address 127.30.2.194:44319)
0923 14:48:50.001735 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:50.001741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:50.001760 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:48:50.001926 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:50.001930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:50.001986 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:50.002085 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:50.002442 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:50.003003 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:50.004403 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:50.004948 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:50.005069 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:50.005119 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:50.005314 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 14:48:50.005377 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260923 14:48:50.005519 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:50.001301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:44319 (local address 127.30.2.193:35121)
0923 14:48:50.001678 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:50.001691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:50.001792 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:50.002139 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:50.002142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:50.002160 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:50.002324 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:50.002333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:50.003122 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:50.003126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:50.004286 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:50.004296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:50.005062 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:50.005066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:50.005157 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:50.005212 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 14:48:50.005299 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 14:48:50.005370 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":127,"thread_start_us":66,"threads_started":1}
I20260923 14:48:50.005676 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:50.001499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:35621 (local address 127.30.2.195:43731)
0923 14:48:50.001618 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:50.001623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:50.001955 (+ 332us) server_negotiation.cc:407] Connection header received
0923 14:48:50.002086 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:50.002090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:50.002173 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:50.002261 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:48:50.002565 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:50.003022 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:50.004464 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:50.004990 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:50.005070 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:50.005112 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:50.005446 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 14:48:50.005514 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:48:50.006134 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:50.001301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:43731 (local address 127.30.2.193:35621)
0923 14:48:50.001826 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:50.001839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:48:50.001965 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:48:50.002294 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:48:50.002296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:48:50.002312 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:48:50.002452 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:50.002459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:50.003163 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:50.003166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:50.004293 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:48:50.004306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:48:50.005215 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:48:50.005218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:48:50.005287 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:50.005303 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 14:48:50.005408 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 14:48:50.005485 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":93,"thread_start_us":80,"threads_started":1}
I20260923 14:48:50.006090 14505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3fce3c1aff4db09600aaa7560150a6" candidate_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ee306f97d3429c913c520e0c702648" is_pre_election: true
I20260923 14:48:50.006392 14505 raft_consensus.cc:2454] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b34f2c2a8a048198ac2aa410f1c6739 in term 0.
I20260923 14:48:50.006775 14637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3fce3c1aff4db09600aaa7560150a6" candidate_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e036e2b041ae43889e4134d09eae6528" is_pre_election: true
I20260923 14:48:50.006893 14298 leader_election.cc:302] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b34f2c2a8a048198ac2aa410f1c6739, e6ee306f97d3429c913c520e0c702648; no voters:
I20260923 14:48:50.007025 14637 raft_consensus.cc:2454] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b34f2c2a8a048198ac2aa410f1c6739 in term 0.
I20260923 14:48:50.007220 14418 raft_consensus.cc:2781] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:48:50.007325 14418 raft_consensus.cc:482] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:48:50.007385 14418 raft_consensus.cc:3037] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:50.008585 14418 raft_consensus.cc:504] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:50.008965 14418 leader_election.cc:288] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 1 election: Requested vote from peers e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731), e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319)
I20260923 14:48:50.009315 14505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3fce3c1aff4db09600aaa7560150a6" candidate_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ee306f97d3429c913c520e0c702648"
I20260923 14:48:50.009315 14637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3fce3c1aff4db09600aaa7560150a6" candidate_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e036e2b041ae43889e4134d09eae6528"
I20260923 14:48:50.009451 14505 raft_consensus.cc:3037] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:50.009451 14637 raft_consensus.cc:3037] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:48:50.010782 14637 raft_consensus.cc:2454] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b34f2c2a8a048198ac2aa410f1c6739 in term 1.
I20260923 14:48:50.010782 14505 raft_consensus.cc:2454] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b34f2c2a8a048198ac2aa410f1c6739 in term 1.
I20260923 14:48:50.011183 14300 leader_election.cc:302] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b34f2c2a8a048198ac2aa410f1c6739, e036e2b041ae43889e4134d09eae6528; no voters:
I20260923 14:48:50.011368 14418 raft_consensus.cc:2781] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:48:50.011574 14418 raft_consensus.cc:686] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 LEADER]: Becoming Leader. State: Replica: 9b34f2c2a8a048198ac2aa410f1c6739, State: Running, Role: LEADER
I20260923 14:48:50.011916 14418 consensus_queue.cc:237] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:48:50.014813 14224 catalog_manager.cc:5495] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b34f2c2a8a048198ac2aa410f1c6739 (127.30.2.193). New cstate: current_term: 1 leader_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } health_report { overall_health: HEALTHY } } }
I20260923 14:48:50.028658 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:50.024637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.30.2.193:46091)
0923 14:48:50.024882 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 14:48:50.024888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:48:50.024906 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:48:50.025008 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:48:50.025010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:48:50.025064 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:48:50.025188 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:48:50.025486 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:50.026005 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:50.027054 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:48:50.027285 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:48:50.027408 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:48:50.027725 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:48:50.028199 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:48:50.028338 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 14:48:50.028431 (+ 93us) server_negotiation.cc:299] Negotiation successful
0923 14:48:50.028490 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260923 14:48:50.032524 14637 raft_consensus.cc:1261] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528 [term 1 FOLLOWER]: Refusing update from remote peer 9b34f2c2a8a048198ac2aa410f1c6739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:50.032526 14505 raft_consensus.cc:1261] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648 [term 1 FOLLOWER]: Refusing update from remote peer 9b34f2c2a8a048198ac2aa410f1c6739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:48:50.033221 14703 consensus_queue.cc:1035] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:48:50.033412 14418 consensus_queue.cc:1035] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:48:50.041851 14708 mvcc.cc:205] Tried to move back new op lower bound from 7332556513406550016 to 7332556513331482624. Current Snapshot: MvccSnapshot[applied={T|T < 7332556513406550016}]
I20260923 14:48:50.043552 14709 mvcc.cc:205] Tried to move back new op lower bound from 7332556513406550016 to 7332556513331482624. Current Snapshot: MvccSnapshot[applied={T|T < 7332556513406550016}]
I20260923 14:48:50.049700 14224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:48:50.049889 14224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:48:50.052263 14224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Sending DeleteTablet for 3 replicas of tablet eb3fce3c1aff4db09600aaa7560150a6
I20260923 14:48:50.052986 14617 tablet_service.cc:1514] Processing DeleteTablet for tablet eb3fce3c1aff4db09600aaa7560150a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:50 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:39970
I20260923 14:48:50.053041 14485 tablet_service.cc:1514] Processing DeleteTablet for tablet eb3fce3c1aff4db09600aaa7560150a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:50 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:49806
I20260923 14:48:50.053496 14718 tablet_replica.cc:330] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528: stopping tablet replica
I20260923 14:48:50.053685 14346 tablet_service.cc:1514] Processing DeleteTablet for tablet eb3fce3c1aff4db09600aaa7560150a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:48:50 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:39866
I20260923 14:48:50.053771 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 14262
I20260923 14:48:50.055017 14718 raft_consensus.cc:2229] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:48:50.055322 14718 raft_consensus.cc:2258] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:50.056094 14717 tablet_replica.cc:330] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648: stopping tablet replica
I20260923 14:48:50.056391 14717 raft_consensus.cc:2229] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:48:50.056681 14717 raft_consensus.cc:2258] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:48:50.056799 14718 ts_tablet_manager.cc:1905] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:50.057976 14717 ts_tablet_manager.cc:1905] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:48:50.059533 14718 ts_tablet_manager.cc:1918] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:50.059638 14718 log.cc:1196] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/eb3fce3c1aff4db09600aaa7560150a6
I20260923 14:48:50.060029 14718 ts_tablet_manager.cc:1939] T eb3fce3c1aff4db09600aaa7560150a6 P e036e2b041ae43889e4134d09eae6528: Deleting consensus metadata
I20260923 14:48:50.060736 14717 ts_tablet_manager.cc:1918] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:48:50.060808 14717 log.cc:1196] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/eb3fce3c1aff4db09600aaa7560150a6
I20260923 14:48:50.061062 14717 ts_tablet_manager.cc:1939] T eb3fce3c1aff4db09600aaa7560150a6 P e6ee306f97d3429c913c520e0c702648: Deleting consensus metadata
I20260923 14:48:50.061970 14210 catalog_manager.cc:4841] TS e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731): tablet eb3fce3c1aff4db09600aaa7560150a6 (table test-table [id=ef2d81a402e64f8cae4d0768effd252b]) successfully deleted
W20260923 14:48:50.062351 14210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:46091 (error 108)
I20260923 14:48:50.062559 14208 catalog_manager.cc:4841] TS e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319): tablet eb3fce3c1aff4db09600aaa7560150a6 (table test-table [id=ef2d81a402e64f8cae4d0768effd252b]) successfully deleted
I20260923 14:48:50.062660 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 14416
I20260923 14:48:50.064631 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:48:50.064370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:46091 (local address 127.0.0.1:39888)
0923 14:48:50.064496 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 14:48:50.064566 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:46091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 14:48:50.064777 14210 catalog_manager.cc:4584] TS 9b34f2c2a8a048198ac2aa410f1c6739 (127.30.2.193:46091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb3fce3c1aff4db09600aaa7560150a6: Network error: Client connection negotiation failed: client connection to 127.30.2.193:46091: connect: Connection refused (error 111)
I20260923 14:48:50.068884 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 14554
I20260923 14:48:50.074930 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 14191
I20260923 14:49:10.082329 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45543
--webserver_interface=127.30.2.254
--webserver_port=40745
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:45543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:10.198028 14720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:10.198400 14720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:10.198441 14720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:10.202322 14720 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 14:49:10.202366 14720 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 14:49:10.202382 14720 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 14:49:10.202395 14720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:10.202412 14720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:10.202430 14720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:10.202445 14720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:10.202461 14720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:10.207511 14720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:45543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=40745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:10.208536 14720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:10.209496 14720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:10.218473 14726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:10.218513 14728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:10.218554 14720 server_base.cc:1034] running on GCE node
W20260923 14:49:10.218508 14725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:10.219211 14720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:10.220389 14720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:10.221580 14720 hybrid_clock.cc:648] HybridClock initialized: now 1790174950221561 us; error 28 us; skew 500 ppm
Sep 23 14:49:10 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174950, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:10.224471 14720 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:49:10.225303 14720 webserver.cc:460] Webserver started at http://127.30.2.254:40745/ using document root <none> and password file <none>
I20260923 14:49:10.225915 14720 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:10.226004 14720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:10.229736 14720 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260923 14:49:10.232044 14735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:10.233254 14720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260923 14:49:10.233420 14720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "c3f5ca6ab39c4fd88e0510234e5757bd"
format_stamp: "Formatted at 2026-09-23 14:48:27 on dist-test-slave-wjf2"
server_key: "cbd60ad78662dc59adbb5b3f6f188669"
server_key_iv: "e2dc38bd7e9f7584c78197a2aa6d632a"
server_key_version: "encryptionkey@0"
I20260923 14:49:10.233944 14720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:10.257130 14720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:10.260082 14720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:10.260280 14720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:10.267190 14720 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:45543
I20260923 14:49:10.267511 14787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:45543 every 8 connection(s)
I20260923 14:49:10.268281 14720 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:10.269068 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 14720
I20260923 14:49:10.269510 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:46091
--local_ip_for_outbound_sockets=127.30.2.193
--tserver_master_addrs=127.30.2.254:45543
--webserver_port=43957
--webserver_interface=127.30.2.193
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:49:10.273638 14788 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:49:10.276409 14788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Bootstrap starting.
I20260923 14:49:10.301527 14788 log.cc:824] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:10.317509 14788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:49:10.318003 14788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Bootstrap complete.
I20260923 14:49:10.323928 14788 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:49:10.324365 14788 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f5ca6ab39c4fd88e0510234e5757bd, State: Initialized, Role: FOLLOWER
I20260923 14:49:10.324998 14788 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:49:10.325176 14788 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:10.325301 14788 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:10.325440 14788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 2 FOLLOWER]: Advancing to term 3
I20260923 14:49:10.327414 14788 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:49:10.327739 14788 leader_election.cc:302] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f5ca6ab39c4fd88e0510234e5757bd; no voters:
I20260923 14:49:10.328270 14788 leader_election.cc:288] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 14:49:10.328429 14792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 3 FOLLOWER]: Leader election won for term 3
I20260923 14:49:10.328680 14792 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [term 3 LEADER]: Becoming Leader. State: Replica: c3f5ca6ab39c4fd88e0510234e5757bd, State: Running, Role: LEADER
I20260923 14:49:10.328996 14792 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } }
I20260923 14:49:10.329514 14788 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:10.329972 14794 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f5ca6ab39c4fd88e0510234e5757bd. Latest consensus state: current_term: 3 leader_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } } }
I20260923 14:49:10.330087 14794 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:10.330482 14793 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f5ca6ab39c4fd88e0510234e5757bd" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45543 } } }
I20260923 14:49:10.330598 14793 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:10.332127 14796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:10.336632 14796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:10.337163 14796 catalog_manager.cc:1259] Loaded cluster ID: cff97b2b81ec42e2a8c059aadcda5c78
I20260923 14:49:10.337235 14796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:10.339195 14796 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:10.339833 14796 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Loaded TSK: 0
I20260923 14:49:10.339922 14796 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3f5ca6ab39c4fd88e0510234e5757bd: Loaded TSK: 1
F20260923 14:49:10.340516 14796 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff1776b09ad google::LogMessage::Fail() at ??:0
@ 0x7ff1776b388a google::LogMessage::SendToLog() at ??:0
@ 0x7ff1776b04de google::LogMessage::Flush() at ??:0
@ 0x7ff1776b1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff17a37418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff17a36cf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff1820157bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff181fef99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff181ff0178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7ff182031df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff18205082b std::function<>::operator()() at ??:0
@ 0x7ff181fefb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff181ff0f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff181fed579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff182031640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff1799023d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff179902c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff1799043f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff1798f3512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff17af376db start_thread at ??:0
@ 0x7ff1761ae61f clone at ??:0
W20260923 14:49:10.434024 14790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:10.434382 14790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:10.434463 14790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:10.441002 14790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:10.441082 14790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:10.441184 14790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:49:10.449077 14790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:46091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=43957
--tserver_master_addrs=127.30.2.254:45543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:10.450513 14790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:10.451928 14790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:10.463476 14835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:10.463514 14836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:10.463505 14838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:10.463753 14790 server_base.cc:1034] running on GCE node
I20260923 14:49:10.464318 14790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:10.464999 14790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:10.466194 14790 hybrid_clock.cc:648] HybridClock initialized: now 1790174950466162 us; error 50 us; skew 500 ppm
Sep 23 14:49:10 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174950, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:10.469182 14790 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:49:10.470119 14790 webserver.cc:460] Webserver started at http://127.30.2.193:43957/ using document root <none> and password file <none>
I20260923 14:49:10.470726 14790 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:10.470786 14790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:10.474807 14790 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:49:10.477145 14845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:10.478499 14790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 14:49:10.478638 14790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "9b34f2c2a8a048198ac2aa410f1c6739"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "5ae5461856cbd1188d416b80ed8e033e"
server_key_iv: "06c49f1591d2224c58096ac9ccd6dbe8"
server_key_version: "encryptionkey@0"
I20260923 14:49:10.479172 14790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:10.495677 14790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:10.498888 14790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:10.499100 14790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:10.499722 14790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:10.500900 14852 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 14:49:10.506844 14790 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 14:49:10.507027 14790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260923 14:49:10.507113 14790 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 14:49:10.508886 14790 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 14:49:10.508961 14790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260923 14:49:10.509269 14852 tablet_bootstrap.cc:492] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: Bootstrap starting.
I20260923 14:49:10.519830 14790 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:46091
I20260923 14:49:10.520076 14959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:46091 every 8 connection(s)
I20260923 14:49:10.520911 14790 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:49:10.530529 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 14790
I20260923 14:49:10.531002 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:44319
--local_ip_for_outbound_sockets=127.30.2.194
--tserver_master_addrs=127.30.2.254:45543
--webserver_port=39251
--webserver_interface=127.30.2.194
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:49:10.531343 14852 log.cc:824] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:10.540724 14852 tablet_bootstrap.cc:492] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:49:10.541204 14852 tablet_bootstrap.cc:492] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: Bootstrap complete.
I20260923 14:49:10.542724 14852 ts_tablet_manager.cc:1397] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.005s
I20260923 14:49:10.545979 14852 raft_consensus.cc:348] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:49:10.546545 14852 raft_consensus.cc:729] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b34f2c2a8a048198ac2aa410f1c6739, State: Initialized, Role: FOLLOWER
I20260923 14:49:10.547084 14852 consensus_queue.cc:260] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:49:10.548163 14852 ts_tablet_manager.cc:1428] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260923 14:49:10.585832 14962 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:49:10.522989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:40433)
0923 14:49:10.523507 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:10.523561 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:10.524426 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:10.584863 (+ 60437us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":155,"threads_started":1}
W20260923 14:49:10.586364 14960 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:45543 (0 consecutive failures): Network error: Failed to ping master at 127.30.2.254:45543: Client connection negotiation failed: client connection to 127.30.2.254:45543: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 14:49:10.587162 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:10.586841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:59849)
0923 14:49:10.587007 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:10.587089 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 14:49:10.587971 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:10.587723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:45921)
0923 14:49:10.587868 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:10.587917 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 14:49:10.588186 14960 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:49:10.645829 14964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:10.646060 14964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:10.646100 14964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:10.650024 14964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:10.650076 14964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:10.650131 14964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:49:10.655452 14964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:44319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=39251
--tserver_master_addrs=127.30.2.254:45543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:10.656543 14964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:10.657648 14964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:10.667634 14970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:10.667644 14973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:10.667683 14971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:10.667912 14964 server_base.cc:1034] running on GCE node
I20260923 14:49:10.668375 14964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:10.669013 14964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:10.670233 14964 hybrid_clock.cc:648] HybridClock initialized: now 1790174950670205 us; error 42 us; skew 500 ppm
Sep 23 14:49:10 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174950, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:10.673061 14964 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:49:10.673884 14964 webserver.cc:460] Webserver started at http://127.30.2.194:39251/ using document root <none> and password file <none>
I20260923 14:49:10.674441 14964 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:10.674539 14964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:10.678339 14964 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:49:10.680469 14980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:10.681716 14964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:49:10.681876 14964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "e6ee306f97d3429c913c520e0c702648"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "6ea1fdf1cb642bdf20d54ba1b9f4b18f"
server_key_iv: "c3d0384e97f44dfafaddff263dedcd42"
server_key_version: "encryptionkey@0"
I20260923 14:49:10.682485 14964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:10.704463 14964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:10.707973 14964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:10.708186 14964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:10.708822 14964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:10.709676 14964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:10.709748 14964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:10.709824 14964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:10.709867 14964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:10.718847 14964 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:44319
I20260923 14:49:10.718907 15093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:44319 every 8 connection(s)
I20260923 14:49:10.719905 14964 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:49:10.721040 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 14964
I20260923 14:49:10.721503 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:43731
--local_ip_for_outbound_sockets=127.30.2.195
--tserver_master_addrs=127.30.2.254:45543
--webserver_port=37651
--webserver_interface=127.30.2.195
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:49:10.724040 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:10.722354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:47127)
0923 14:49:10.723168 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:10.723270 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":149,"threads_started":1}
W20260923 14:49:10.724558 15094 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:45543 (0 consecutive failures): Network error: Failed to ping master at 127.30.2.254:45543: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
I20260923 14:49:10.725204 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:10.724983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:57071)
0923 14:49:10.725102 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:10.725148 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 14:49:10.725975 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:10.725674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:36323)
0923 14:49:10.725828 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:10.725895 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 14:49:10.726171 15094 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:49:10.833238 15096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:10.833508 15096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:10.833549 15096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:10.837574 15096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:10.837626 15096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:10.837720 15096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:49:10.842887 15096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:43731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=37651
--tserver_master_addrs=127.30.2.254:45543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:10.844066 15096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:10.845149 15096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:49:10.855154 15096 server_base.cc:1034] running on GCE node
W20260923 14:49:10.855149 15103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:10.855103 15102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:10.855103 15105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:10.855979 15096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:10.856583 15096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:10.857734 15096 hybrid_clock.cc:648] HybridClock initialized: now 1790174950857709 us; error 35 us; skew 500 ppm
Sep 23 14:49:10 dist-test-slave-wjf2 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174950, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:10.860698 15096 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:49:10.861579 15096 webserver.cc:460] Webserver started at http://127.30.2.195:37651/ using document root <none> and password file <none>
I20260923 14:49:10.862109 15096 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:10.862174 15096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:10.866097 15096 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260923 14:49:10.868301 15112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:10.869585 15096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:10.869691 15096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "e036e2b041ae43889e4134d09eae6528"
format_stamp: "Formatted at 2026-09-23 14:48:28 on dist-test-slave-wjf2"
server_key: "74d8779d4a8d76702656be0e3644b0b0"
server_key_iv: "8eda4e5a6af93f5c648e3ecafd2a5794"
server_key_version: "encryptionkey@0"
I20260923 14:49:10.870326 15096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:10.880838 15096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:10.884378 15096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:10.884575 15096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:10.885179 15096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:10.886057 15096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:10.886109 15096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:10.886152 15096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:10.886206 15096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:10.894979 15096 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:43731
I20260923 14:49:10.895037 15226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:43731 every 8 connection(s)
I20260923 14:49:10.896130 15096 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:49:10.898500 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 15096
I20260923 14:49:10.900113 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:10.898378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:56261)
0923 14:49:10.899123 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:10.899254 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":143,"threads_started":1}
W20260923 14:49:10.900576 15227 heartbeater.cc:644] Failed to heartbeat to 127.30.2.254:45543 (0 consecutive failures): Network error: Failed to ping master at 127.30.2.254:45543: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
I20260923 14:49:10.901172 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:10.900970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:51683)
0923 14:49:10.901073 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:10.901120 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 14:49:10.901852 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:10.901662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:40381)
0923 14:49:10.901766 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:10.901807 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 14:49:10.902112 15227 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 14:49:11.589131 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:11.588709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:44511)
0923 14:49:11.588964 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:11.589019 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260923 14:49:11.727078 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:11.726698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:51809)
0923 14:49:11.726928 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:11.726979 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260923 14:49:11.903116 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:11.902664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:50833)
0923 14:49:11.902947 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:11.903009 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":69,"threads_started":1}
I20260923 14:49:12.269768 15234 raft_consensus.cc:482] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:49:12.270077 15234 raft_consensus.cc:504] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:49:12.271476 15234 leader_election.cc:288] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731), e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319)
Sep 23 14:49:12 dist-test-slave-wjf2 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174950, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.194@KRBTEST.COM
Sep 23 14:49:12 dist-test-slave-wjf2 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174950, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.195@KRBTEST.COM
I20260923 14:49:12.286329 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:12.272054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:43813 (local address 127.30.2.194:44319)
0923 14:49:12.272504 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:12.272523 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:12.272580 (+ 57us) server_negotiation.cc:407] Connection header received
0923 14:49:12.273523 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:12.273530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:12.273858 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:12.274195 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:12.275563 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:12.276727 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:12.277883 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:12.278441 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:12.278628 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:12.281506 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:12.281510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:12.281522 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:12.281548 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:12.283802 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:12.284398 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:12.284404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:12.284409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:12.284529 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:12.284869 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:12.284873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:12.284875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:12.285257 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:12.285405 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:12.285919 (+ 514us) server_negotiation.cc:299] Negotiation successful
0923 14:49:12.286167 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":89,"threads_started":1}
I20260923 14:49:12.286351 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:12.271750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:44319 (local address 127.30.2.193:43813)
0923 14:49:12.272153 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:12.272167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:12.272268 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:12.274243 (+ 1975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:12.274249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:12.274856 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:12.275350 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:12.275368 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:12.276874 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:12.276876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:12.277758 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:12.277767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:12.278578 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:12.278582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:12.278761 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:12.279459 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:49:12.279471 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:12.281245 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:12.284026 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:12.284030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:12.284041 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:12.284278 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:12.284636 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:12.284639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:12.284641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:12.284772 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:12.285378 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:12.285391 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:12.285616 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 14:49:12.285799 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 14:49:12.286125 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":82,"spinlock_wait_cycles":26752,"thread_start_us":90,"threads_started":1}
I20260923 14:49:12.286721 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:12.271678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:43731 (local address 127.30.2.193:47563)
0923 14:49:12.272076 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:12.272093 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:12.272230 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:12.274245 (+ 2015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:12.274251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:12.274847 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:12.275349 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:12.275368 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:12.276784 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:12.276792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:12.277764 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:12.277777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:12.278506 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:12.278514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:12.278770 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:12.279398 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:49:12.279417 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:12.281656 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:12.284570 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:12.284578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:12.284580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:12.284818 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:12.285219 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:12.285223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:12.285225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:12.285329 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:12.286077 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:12.286081 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:12.286196 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 14:49:12.286349 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 14:49:12.286503 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":168,"thread_start_us":119,"threads_started":1}
I20260923 14:49:12.287032 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:12.271891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:47563 (local address 127.30.2.195:43731)
0923 14:49:12.272359 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:12.272388 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:12.272431 (+ 43us) server_negotiation.cc:407] Connection header received
0923 14:49:12.273481 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:12.273488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:12.273869 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:12.274213 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:12.275538 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:12.276627 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:12.277926 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:12.278370 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:12.278491 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:12.281875 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:12.281880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:12.281892 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:12.281923 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:12.284418 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:12.284950 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:12.284957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:12.284962 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:12.285094 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:12.285470 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:12.285480 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:12.285481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:12.285934 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:12.286116 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:12.286513 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 14:49:12.286850 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":120,"threads_started":1}
W20260923 14:49:12.289433 14846 leader_election.cc:341] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319): Not found: Tablet not found: eb3fce3c1aff4db09600aaa7560150a6
W20260923 14:49:12.289623 14848 leader_election.cc:341] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731): Not found: Tablet not found: eb3fce3c1aff4db09600aaa7560150a6
I20260923 14:49:12.289772 14848 leader_election.cc:302] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b34f2c2a8a048198ac2aa410f1c6739; no voters: e036e2b041ae43889e4134d09eae6528, e6ee306f97d3429c913c520e0c702648
I20260923 14:49:12.290071 15234 raft_consensus.cc:2726] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:49:12.590160 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:12.589866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:49837)
0923 14:49:12.590018 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:12.590072 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 14:49:12.728336 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:12.727944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:44401)
0923 14:49:12.728181 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:12.728233 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260923 14:49:12.904508 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:12.904074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:58319)
0923 14:49:12.904357 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:12.904414 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260923 14:49:13.591315 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:13.590904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:34773)
0923 14:49:13.591165 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:13.591219 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260923 14:49:13.729423 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:13.729060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:47271)
0923 14:49:13.729286 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:13.729337 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260923 14:49:13.905651 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:13.905219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:53523)
0923 14:49:13.905486 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:13.905542 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
I20260923 14:49:14.275900 15244 raft_consensus.cc:482] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:49:14.276031 15244 raft_consensus.cc:504] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:49:14.276422 15244 leader_election.cc:288] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731), e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319)
W20260923 14:49:14.277248 14846 leader_election.cc:341] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319): Not found: Tablet not found: eb3fce3c1aff4db09600aaa7560150a6
W20260923 14:49:14.277334 14848 leader_election.cc:341] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731): Not found: Tablet not found: eb3fce3c1aff4db09600aaa7560150a6
I20260923 14:49:14.277456 14848 leader_election.cc:302] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b34f2c2a8a048198ac2aa410f1c6739; no voters: e036e2b041ae43889e4134d09eae6528, e6ee306f97d3429c913c520e0c702648
I20260923 14:49:14.277611 15244 raft_consensus.cc:2726] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:49:14.592684 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:14.592254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:47805)
0923 14:49:14.592514 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:14.592569 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260923 14:49:14.730624 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:14.730188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:60603)
0923 14:49:14.730452 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:14.730513 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260923 14:49:14.906836 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:14.906439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:52983)
0923 14:49:14.906693 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:14.906747 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260923 14:49:15.594228 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:15.593702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:44555)
0923 14:49:15.594049 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:15.594113 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":87,"threads_started":1}
I20260923 14:49:15.732019 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:15.731478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:59665)
0923 14:49:15.731765 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:15.731909 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260923 14:49:15.908109 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:15.907631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:47081)
0923 14:49:15.907950 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:15.908011 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":76,"threads_started":1}
I20260923 14:49:16.595547 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:16.595105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:55361)
0923 14:49:16.595385 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:16.595443 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260923 14:49:16.733171 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:16.732738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:53435)
0923 14:49:16.733029 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:16.733079 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":97,"threads_started":1}
I20260923 14:49:16.909301 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:16.908873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:48283)
0923 14:49:16.909147 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:16.909203 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260923 14:49:17.073719 15254 raft_consensus.cc:482] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:49:17.073853 15254 raft_consensus.cc:504] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:49:17.074322 15254 leader_election.cc:288] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731), e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319)
W20260923 14:49:17.075145 14848 leader_election.cc:341] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731): Not found: Tablet not found: eb3fce3c1aff4db09600aaa7560150a6
W20260923 14:49:17.075287 14846 leader_election.cc:341] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319): Not found: Tablet not found: eb3fce3c1aff4db09600aaa7560150a6
I20260923 14:49:17.075392 14846 leader_election.cc:302] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b34f2c2a8a048198ac2aa410f1c6739; no voters: e036e2b041ae43889e4134d09eae6528, e6ee306f97d3429c913c520e0c702648
I20260923 14:49:17.075572 15254 raft_consensus.cc:2726] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:49:17.596778 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:17.596379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:37383)
0923 14:49:17.596626 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:17.596681 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260923 14:49:17.734342 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:17.733907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:35643)
0923 14:49:17.734200 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:17.734249 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":92,"threads_started":1}
I20260923 14:49:17.910588 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:17.910131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:40441)
0923 14:49:17.910441 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:17.910496 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":91,"threads_started":1}
I20260923 14:49:18.597872 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:18.597482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:40685)
0923 14:49:18.597723 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:18.597775 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260923 14:49:18.735426 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:18.735014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:48753)
0923 14:49:18.735277 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:18.735333 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260923 14:49:18.911684 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:18.911299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:59881)
0923 14:49:18.911541 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:18.911591 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260923 14:49:19.599102 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:19.598668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:60439)
0923 14:49:19.598945 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:19.598999 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":101,"threads_started":1}
I20260923 14:49:19.725015 15262 raft_consensus.cc:482] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:49:19.725139 15262 raft_consensus.cc:504] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:49:19.725649 15262 leader_election.cc:288] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731), e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319)
W20260923 14:49:19.726401 14846 leader_election.cc:341] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319): Not found: Tablet not found: eb3fce3c1aff4db09600aaa7560150a6
W20260923 14:49:19.726521 14848 leader_election.cc:341] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731): Not found: Tablet not found: eb3fce3c1aff4db09600aaa7560150a6
I20260923 14:49:19.726629 14848 leader_election.cc:302] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b34f2c2a8a048198ac2aa410f1c6739; no voters: e036e2b041ae43889e4134d09eae6528, e6ee306f97d3429c913c520e0c702648
I20260923 14:49:19.726779 15262 raft_consensus.cc:2726] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:49:19.736531 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:19.736144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:41839)
0923 14:49:19.736397 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:19.736448 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260923 14:49:19.912971 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:19.912506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:42885)
0923 14:49:19.912822 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:19.912876 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":134,"threads_started":1}
I20260923 14:49:20.600553 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:20.600058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:45201)
0923 14:49:20.600365 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:20.600432 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":70,"threads_started":1}
I20260923 14:49:20.737588 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:20.737210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:38523)
0923 14:49:20.737446 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:20.737498 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260923 14:49:20.914250 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:20.913777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:37835)
0923 14:49:20.914096 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:20.914154 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":95,"threads_started":1}
I20260923 14:49:21.275478 15268 raft_consensus.cc:482] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:49:21.275671 15268 raft_consensus.cc:504] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e036e2b041ae43889e4134d09eae6528" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 43731 } } peers { permanent_uuid: "e6ee306f97d3429c913c520e0c702648" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 44319 } } peers { permanent_uuid: "9b34f2c2a8a048198ac2aa410f1c6739" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 46091 } }
I20260923 14:49:21.276175 15268 leader_election.cc:288] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731), e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319)
W20260923 14:49:21.276937 14848 leader_election.cc:341] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e036e2b041ae43889e4134d09eae6528 (127.30.2.195:43731): Not found: Tablet not found: eb3fce3c1aff4db09600aaa7560150a6
W20260923 14:49:21.277024 14846 leader_election.cc:341] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6ee306f97d3429c913c520e0c702648 (127.30.2.194:44319): Not found: Tablet not found: eb3fce3c1aff4db09600aaa7560150a6
I20260923 14:49:21.277117 14846 leader_election.cc:302] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b34f2c2a8a048198ac2aa410f1c6739; no voters: e036e2b041ae43889e4134d09eae6528, e6ee306f97d3429c913c520e0c702648
I20260923 14:49:21.277256 15268 raft_consensus.cc:2726] T eb3fce3c1aff4db09600aaa7560150a6 P 9b34f2c2a8a048198ac2aa410f1c6739 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:49:21.601765 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:21.601338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:36337)
0923 14:49:21.601600 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:21.601655 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260923 14:49:21.738875 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:21.738377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:44957)
0923 14:49:21.738642 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:21.738713 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":109,"threads_started":1}
I20260923 14:49:21.915547 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:21.915092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:35765)
0923 14:49:21.915387 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:21.915447 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
I20260923 14:49:22.602936 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:22.602510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:38245)
0923 14:49:22.602779 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:22.602834 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260923 14:49:22.740080 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:22.739616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:39787)
0923 14:49:22.739923 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:22.739980 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":73,"threads_started":1}
I20260923 14:49:22.916822 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:22.916340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:54513)
0923 14:49:22.916643 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:22.916704 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":106,"threads_started":1}
I20260923 14:49:23.604189 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:23.603718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:53025)
0923 14:49:23.604029 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:23.604079 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":150,"threads_started":1}
I20260923 14:49:23.741325 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:23.740864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:46845)
0923 14:49:23.741138 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:23.741210 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20260923 14:49:23.918011 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:23.917586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:57919)
0923 14:49:23.917856 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:23.917912 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260923 14:49:24.605418 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:24.604964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:33591)
0923 14:49:24.605243 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:24.605311 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20260923 14:49:24.742658 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:24.742212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:32927)
0923 14:49:24.742499 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:24.742558 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
I20260923 14:49:24.919334 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:24.918870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:46711)
0923 14:49:24.919171 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:24.919234 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20260923 14:49:25.607053 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:25.606560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.193:47271)
0923 14:49:25.606878 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:25.606947 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260923 14:49:25.744297 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:25.743770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.194:56021)
0923 14:49:25.744130 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:25.744194 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":97,"threads_started":1}
I20260923 14:49:25.909030 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 14790
I20260923 14:49:25.915573 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 14964
I20260923 14:49:25.920810 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:25.920371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:45543 (local address 127.30.2.195:56679)
0923 14:49:25.920642 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:25.920703 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.254:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
I20260923 14:49:25.921319 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 15096
2026-09-23T14:49:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59269 ms)
[----------] 30 tests from SecurityITest (242574 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:25 dist-test-slave-wjf2 krb5kdc[15286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:25 dist-test-slave-wjf2 krb5kdc[15286](info): set up 2 sockets
Sep 23 14:49:25 dist-test-slave-wjf2 krb5kdc[15286](info): commencing operation
krb5kdc: starting...
W20260923 14:49:26.997026 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:27 dist-test-slave-wjf2 krb5kdc[15286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:49:27.123652 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:40041
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:40041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:27.234663 15299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:27.234977 15299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:27.235052 15299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:27.239295 15299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:27.239369 15299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:27.239405 15299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:27.239442 15299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:27.239502 15299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:27.245314 15299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:40041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:40041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:27.246850 15299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:27.248193 15299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:27.257922 15305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:27.257939 15304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:27.258378 15307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:27.258656 15299 server_base.cc:1034] running on GCE node
I20260923 14:49:27.259227 15299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:27.260403 15299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:27.261564 15299 hybrid_clock.cc:648] HybridClock initialized: now 1790174967261550 us; error 25 us; skew 500 ppm
Sep 23 14:49:27 dist-test-slave-wjf2 krb5kdc[15286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:27.264453 15299 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:49:27.265398 15299 webserver.cc:460] Webserver started at http://127.30.2.254:41141/ using document root <none> and password file <none>
I20260923 14:49:27.266048 15299 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:27.266115 15299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:27.266357 15299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:27.268672 15299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "abd792ae3c24485fa4b2810b6c9f0f5f"
format_stamp: "Formatted at 2026-09-23 14:49:27 on dist-test-slave-wjf2"
server_key: "e587f44513c8e6fe49fff0514f157274"
server_key_iv: "bb91026c314c83dbcd347b0b9b6e2d45"
server_key_version: "encryptionkey@0"
I20260923 14:49:27.269331 15299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "abd792ae3c24485fa4b2810b6c9f0f5f"
format_stamp: "Formatted at 2026-09-23 14:49:27 on dist-test-slave-wjf2"
server_key: "e587f44513c8e6fe49fff0514f157274"
server_key_iv: "bb91026c314c83dbcd347b0b9b6e2d45"
server_key_version: "encryptionkey@0"
I20260923 14:49:27.273372 15299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:49:27.275982 15315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:27.277237 15299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:49:27.277390 15299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "abd792ae3c24485fa4b2810b6c9f0f5f"
format_stamp: "Formatted at 2026-09-23 14:49:27 on dist-test-slave-wjf2"
server_key: "e587f44513c8e6fe49fff0514f157274"
server_key_iv: "bb91026c314c83dbcd347b0b9b6e2d45"
server_key_version: "encryptionkey@0"
I20260923 14:49:27.277506 15299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:27.309243 15299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:27.312621 15299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:27.312844 15299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:27.320628 15299 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:40041
I20260923 14:49:27.320653 15367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:40041 every 8 connection(s)
I20260923 14:49:27.321624 15299 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:27.324492 15368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:27.330025 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f: Bootstrap starting.
I20260923 14:49:27.330052 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 15299
I20260923 14:49:27.330209 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:27.330511 30731 external_mini_cluster.cc:1428] Setting key cfadde6f39e2ccd463d5da7b653f585e
I20260923 14:49:27.332854 15368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:27.333734 15368 log.cc:824] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:27.335948 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f: No bootstrap required, opened a new log
Sep 23 14:49:27 dist-test-slave-wjf2 krb5kdc[15286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:49:27.340009 15368 raft_consensus.cc:348] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd792ae3c24485fa4b2810b6c9f0f5f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40041 } }
I20260923 14:49:27.340265 15368 raft_consensus.cc:374] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:27.340353 15368 raft_consensus.cc:729] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd792ae3c24485fa4b2810b6c9f0f5f, State: Initialized, Role: FOLLOWER
I20260923 14:49:27.340857 15368 consensus_queue.cc:260] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd792ae3c24485fa4b2810b6c9f0f5f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40041 } }
I20260923 14:49:27.341033 15368 raft_consensus.cc:388] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:27.341112 15368 raft_consensus.cc:482] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:27.341251 15368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:27.342563 15368 raft_consensus.cc:504] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd792ae3c24485fa4b2810b6c9f0f5f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40041 } }
I20260923 14:49:27.342957 15368 leader_election.cc:302] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abd792ae3c24485fa4b2810b6c9f0f5f; no voters:
I20260923 14:49:27.343396 15368 leader_election.cc:288] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:27.343494 15373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:27.343690 15373 raft_consensus.cc:686] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [term 1 LEADER]: Becoming Leader. State: Replica: abd792ae3c24485fa4b2810b6c9f0f5f, State: Running, Role: LEADER
I20260923 14:49:27.344050 15373 consensus_queue.cc:237] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd792ae3c24485fa4b2810b6c9f0f5f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40041 } }
I20260923 14:49:27.344754 15368 sys_catalog.cc:564] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:27.345675 15374 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abd792ae3c24485fa4b2810b6c9f0f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd792ae3c24485fa4b2810b6c9f0f5f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40041 } } }
I20260923 14:49:27.345789 15374 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:27.346167 15375 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader abd792ae3c24485fa4b2810b6c9f0f5f. Latest consensus state: current_term: 1 leader_uuid: "abd792ae3c24485fa4b2810b6c9f0f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd792ae3c24485fa4b2810b6c9f0f5f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 40041 } } }
I20260923 14:49:27.346315 15375 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:27.347148 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:27.331991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.30.2.254:40041)
0923 14:49:27.332414 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:27.332432 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:27.332473 (+ 41us) server_negotiation.cc:407] Connection header received
0923 14:49:27.333364 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:27.333388 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:27.333815 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:27.334196 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:27.335023 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:27.336473 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:27.337329 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:27.338040 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:27.338200 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:27.340812 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:27.340819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:27.340834 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:27.340866 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:27.343067 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:27.343583 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:27.343591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:27.343599 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:27.343811 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:27.344109 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:27.344112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:27.344114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:27.344594 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:27.345457 (+ 863us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:27.345620 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 14:49:27.345948 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
W20260923 14:49:27.350070 15389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:49:27.350157 15389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:49:27.350243 15382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:27.350917 15382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:27.355919 15382 catalog_manager.cc:1347] Generated new cluster ID: 5acd40ec869a4236a622946c9b689789
I20260923 14:49:27.355978 15382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:27.369143 15382 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:27.370302 15382 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:27.378063 15382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f: Generated new TSK 0
I20260923 14:49:27.378788 15382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:49:27.413046 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40041
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:27.533627 15394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:27.533893 15394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:27.533938 15394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:27.539639 15394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:27.539701 15394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:27.539825 15394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:49:27.545210 15394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:27.546371 15394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:27.547489 15394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:27.557682 15400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:27.557689 15402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:27.557720 15399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:27.558009 15394 server_base.cc:1034] running on GCE node
I20260923 14:49:27.558390 15394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:27.559005 15394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:27.560187 15394 hybrid_clock.cc:648] HybridClock initialized: now 1790174967560156 us; error 48 us; skew 500 ppm
Sep 23 14:49:27 dist-test-slave-wjf2 krb5kdc[15286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:27.562731 15394 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:49:27.563627 15394 webserver.cc:460] Webserver started at http://127.30.2.193:40775/ using document root <none> and password file <none>
I20260923 14:49:27.564314 15394 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:27.564400 15394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:27.564633 15394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:27.566700 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "aa175fd231e541f5a11d05cdc5633eaf"
format_stamp: "Formatted at 2026-09-23 14:49:27 on dist-test-slave-wjf2"
server_key: "e943a58fa9cc28dac457f9a1d02b9e30"
server_key_iv: "bc972899a0bfb450510ccf6da3917112"
server_key_version: "encryptionkey@0"
I20260923 14:49:27.567198 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "aa175fd231e541f5a11d05cdc5633eaf"
format_stamp: "Formatted at 2026-09-23 14:49:27 on dist-test-slave-wjf2"
server_key: "e943a58fa9cc28dac457f9a1d02b9e30"
server_key_iv: "bc972899a0bfb450510ccf6da3917112"
server_key_version: "encryptionkey@0"
I20260923 14:49:27.570817 15394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 14:49:27.573161 15409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:27.574324 15394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:49:27.574440 15394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "aa175fd231e541f5a11d05cdc5633eaf"
format_stamp: "Formatted at 2026-09-23 14:49:27 on dist-test-slave-wjf2"
server_key: "e943a58fa9cc28dac457f9a1d02b9e30"
server_key_iv: "bc972899a0bfb450510ccf6da3917112"
server_key_version: "encryptionkey@0"
I20260923 14:49:27.574532 15394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:27.593145 15394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:27.596151 15394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:27.596345 15394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:27.596871 15394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:27.597661 15394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:27.597728 15394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:27.597790 15394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:27.597841 15394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:27.605806 15394 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:39765
I20260923 14:49:27.605880 15524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:39765 every 8 connection(s)
I20260923 14:49:27.606660 15394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:49:27.609954 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 15394
I20260923 14:49:27.610078 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:49:27.610374 30731 external_mini_cluster.cc:1428] Setting key c3698fa583e602f0ee7dd38bfa01b41a
Sep 23 14:49:27 dist-test-slave-wjf2 krb5kdc[15286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:49:27.619974 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:27.608435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:48125 (local address 127.30.2.254:40041)
0923 14:49:27.608560 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:27.608564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:27.609506 (+ 942us) server_negotiation.cc:407] Connection header received
0923 14:49:27.610493 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:27.610496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:27.610552 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:27.610643 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:27.611976 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:27.612502 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:27.613646 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:27.613873 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:27.613981 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:27.616413 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:27.616417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:27.616420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:27.616450 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:27.617903 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:27.618396 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:27.618401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:27.618403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:27.618482 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:27.618834 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:27.618838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:27.618840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:27.619015 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:27.619118 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:27.619656 (+ 538us) server_negotiation.cc:299] Negotiation successful
0923 14:49:27.619772 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:49:27.620724 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:27.608779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40041 (local address 127.30.2.193:48125)
0923 14:49:27.609379 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:27.609416 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:27.610302 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:27.610820 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:27.610842 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:27.611210 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:27.611764 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:27.611842 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:27.612632 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:27.612635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:27.613491 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:27.613500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:27.614019 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:27.614023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:27.614181 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:27.614757 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:49:27.614772 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:27.616241 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:27.618020 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:27.618025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:27.618040 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:27.618294 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:27.618581 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:27.618584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:27.618586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:27.618725 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:27.619130 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:27.619138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:27.619419 (+ 281us) client_negotiation.cc:769] Sending connection context
0923 14:49:27.619600 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 14:49:27.619955 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":105,"threads_started":1}
I20260923 14:49:27.621958 15525 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
I20260923 14:49:27.622269 15525 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:27.622875 15525 heartbeater.cc:511] Master 127.30.2.254:40041 requested a full tablet report, sending...
I20260923 14:49:27.624496 15332 ts_manager.cc:194] Registered new tserver with Master: aa175fd231e541f5a11d05cdc5633eaf (127.30.2.193:39765)
I20260923 14:49:27.625583 15332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:48125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:49:27.638849 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40041
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:27.748812 15530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:27.749082 15530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:27.749123 15530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:27.753108 15530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:27.753155 15530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:27.753244 15530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:49:27.758399 15530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:27.759387 15530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:27.760465 15530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:27.769856 15535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:27.769992 15530 server_base.cc:1034] running on GCE node
W20260923 14:49:27.769855 15536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:27.769917 15538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:27.770555 15530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:27.771274 15530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:27.772467 15530 hybrid_clock.cc:648] HybridClock initialized: now 1790174967772449 us; error 25 us; skew 500 ppm
Sep 23 14:49:27 dist-test-slave-wjf2 krb5kdc[15286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:27.779713 15530 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:49:27.780738 15530 webserver.cc:460] Webserver started at http://127.30.2.194:40775/ using document root <none> and password file <none>
I20260923 14:49:27.781371 15530 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:27.781436 15530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:27.781705 15530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:27.783972 15530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "2409558ed6b3420fb9910b01743ccc56"
format_stamp: "Formatted at 2026-09-23 14:49:27 on dist-test-slave-wjf2"
server_key: "805c09f18dcfba43fd4c2867f863f6de"
server_key_iv: "0b86b10146507d7f39e9a778f754883a"
server_key_version: "encryptionkey@0"
I20260923 14:49:27.784514 15530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "2409558ed6b3420fb9910b01743ccc56"
format_stamp: "Formatted at 2026-09-23 14:49:27 on dist-test-slave-wjf2"
server_key: "805c09f18dcfba43fd4c2867f863f6de"
server_key_iv: "0b86b10146507d7f39e9a778f754883a"
server_key_version: "encryptionkey@0"
I20260923 14:49:27.788329 15530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 14:49:27.790751 15545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:27.792198 15530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:49:27.792304 15530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "2409558ed6b3420fb9910b01743ccc56"
format_stamp: "Formatted at 2026-09-23 14:49:27 on dist-test-slave-wjf2"
server_key: "805c09f18dcfba43fd4c2867f863f6de"
server_key_iv: "0b86b10146507d7f39e9a778f754883a"
server_key_version: "encryptionkey@0"
I20260923 14:49:27.792389 15530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:27.806680 15530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:27.809934 15530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:27.810104 15530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:27.810640 15530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:27.811493 15530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:27.811555 15530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:27.811605 15530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:27.811637 15530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:27.820722 15530 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:38821
I20260923 14:49:27.820748 15659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:38821 every 8 connection(s)
I20260923 14:49:27.821641 15530 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:49:27.824998 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 15530
I20260923 14:49:27.825116 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:49:27.825484 30731 external_mini_cluster.cc:1428] Setting key aa7623dba7e59069d766024dd249dcf4
Sep 23 14:49:27 dist-test-slave-wjf2 krb5kdc[15286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
I20260923 14:49:27.837003 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:27.823966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:32869 (local address 127.30.2.254:40041)
0923 14:49:27.824096 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:27.824107 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:27.825075 (+ 968us) server_negotiation.cc:407] Connection header received
0923 14:49:27.826246 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:27.826249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:27.826325 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:27.826415 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:27.827961 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:27.828474 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:27.829751 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:27.830187 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:27.830309 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:27.832916 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:27.832923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:27.832928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:27.832965 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:27.834651 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:27.835242 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:27.835247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:27.835252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:27.835349 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:27.835936 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:27.835940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:27.835942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:27.836132 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:27.836212 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:27.836754 (+ 542us) server_negotiation.cc:299] Negotiation successful
0923 14:49:27.836854 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:49:27.838166 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:27.824176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40041 (local address 127.30.2.194:32869)
0923 14:49:27.824880 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:27.824938 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:27.825999 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:27.826562 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:27.826568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:27.827062 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:27.827751 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:27.827806 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:27.828603 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:27.828610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:27.829618 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:27.829628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:27.830363 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:27.830366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:27.830508 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:27.831177 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:49:27.831195 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:27.832708 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:27.834795 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:27.834804 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:27.834823 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:27.835110 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:27.835498 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:27.835501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:27.835503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:27.835726 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:27.836233 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:27.836239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:27.836491 (+ 252us) client_negotiation.cc:769] Sending connection context
0923 14:49:27.836687 (+ 196us) client_negotiation.cc:240] Negotiation successful
0923 14:49:27.837099 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
I20260923 14:49:27.839478 15660 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40041
I20260923 14:49:27.839726 15660 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:27.840469 15660 heartbeater.cc:511] Master 127.30.2.254:40041 requested a full tablet report, sending...
I20260923 14:49:27.841638 15332 ts_manager.cc:194] Registered new tserver with Master: 2409558ed6b3420fb9910b01743ccc56 (127.30.2.194:38821)
I20260923 14:49:27.842278 15332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:32869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:49:27.855261 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40041
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:27.978072 15665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:27.978304 15665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:27.978376 15665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:27.982674 15665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:27.982725 15665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:27.982806 15665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:49:27.987967 15665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:40041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:27.988905 15665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:27.989866 15665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:27.999469 15670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:27.999521 15671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:27.999538 15673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:27.999843 15665 server_base.cc:1034] running on GCE node
I20260923 14:49:28.000262 15665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:28.000867 15665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:28.002053 15665 hybrid_clock.cc:648] HybridClock initialized: now 1790174968002024 us; error 50 us; skew 500 ppm
Sep 23 14:49:28 dist-test-slave-wjf2 krb5kdc[15286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:28.004809 15665 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:49:28.005597 15665 webserver.cc:460] Webserver started at http://127.30.2.195:36951/ using document root <none> and password file <none>
I20260923 14:49:28.006134 15665 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:28.006218 15665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:28.006438 15665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:28.008459 15665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "16088807d03247a4afdb3a4017de907f"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "57b10ec07e3ac808d2914224b1b258a9"
server_key_iv: "3aceaa6faf1d239444d110f464493eaa"
server_key_version: "encryptionkey@0"
I20260923 14:49:28.009008 15665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "16088807d03247a4afdb3a4017de907f"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "57b10ec07e3ac808d2914224b1b258a9"
server_key_iv: "3aceaa6faf1d239444d110f464493eaa"
server_key_version: "encryptionkey@0"
I20260923 14:49:28.012760 15665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:49:28.015194 15680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:28.016496 15665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:28.016628 15665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "16088807d03247a4afdb3a4017de907f"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "57b10ec07e3ac808d2914224b1b258a9"
server_key_iv: "3aceaa6faf1d239444d110f464493eaa"
server_key_version: "encryptionkey@0"
I20260923 14:49:28.016770 15665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:28.029677 15665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:28.033016 15665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:28.033209 15665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:28.033784 15665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:28.034574 15665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:28.034641 15665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:28.034713 15665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:28.034756 15665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:28.044160 15665 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:40003
I20260923 14:49:28.044207 15794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:40003 every 8 connection(s)
I20260923 14:49:28.045135 15665 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:49:28.051297 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 15665
I20260923 14:49:28.051445 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:49:28.051734 30731 external_mini_cluster.cc:1428] Setting key 7d9b24ea5410e222f8bb680e9b987283
Sep 23 14:49:28 dist-test-slave-wjf2 krb5kdc[15286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:49:28.058231 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.046938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:40813 (local address 127.30.2.254:40041)
0923 14:49:28.047093 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:28.047097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:28.048113 (+ 1016us) server_negotiation.cc:407] Connection header received
0923 14:49:28.048928 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:28.048930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:28.048976 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:28.049048 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:28.050127 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.050664 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.051889 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.052175 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.052300 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.054850 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:28.054854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:28.054857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:28.054885 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:28.056225 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:28.056737 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:28.056741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:28.056743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:28.056823 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:28.057173 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:28.057176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:28.057178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:28.057361 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:28.057465 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:28.057938 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 14:49:28.058067 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 14:49:28.058956 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.047313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:40041 (local address 127.30.2.195:40813)
0923 14:49:28.047991 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:28.048031 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:28.048753 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:28.049204 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:28.049210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:28.049527 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:28.049971 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.049984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.050768 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.050771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.051731 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.051741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.052354 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.052358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.052523 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.053036 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:49:28.053052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:28.054678 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:28.056361 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:28.056367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:28.056381 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:28.056630 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:28.056929 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:28.056932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:28.056934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:28.057066 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:28.057464 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:28.057471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:28.057707 (+ 236us) client_negotiation.cc:769] Sending connection context
0923 14:49:28.057891 (+ 184us) client_negotiation.cc:240] Negotiation successful
0923 14:49:28.058192 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":108,"threads_started":1}
I20260923 14:49:28.060065 15795 heartbeater.cc:347] Connected to a master server at 127.30.2.254:40041
I20260923 14:49:28.060341 15795 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:28.060855 15795 heartbeater.cc:511] Master 127.30.2.254:40041 requested a full tablet report, sending...
I20260923 14:49:28.061905 15332 ts_manager.cc:194] Registered new tserver with Master: 16088807d03247a4afdb3a4017de907f (127.30.2.195:40003)
I20260923 14:49:28.062414 15332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:40813
I20260923 14:49:28.065804 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:49:28.074563 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.067674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.30.2.254:40041)
0923 14:49:28.067822 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:28.067826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:28.068012 (+ 186us) server_negotiation.cc:407] Connection header received
0923 14:49:28.068106 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:28.068108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:28.068142 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:28.068204 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:28.068842 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.069310 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.070085 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.070276 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.070363 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.071901 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:28.071904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:28.071906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:28.071927 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:28.073277 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:28.073639 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:28.073641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:28.073642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:28.073708 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:28.073928 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:28.073930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:28.073931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:28.074106 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:28.074200 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:28.074382 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 14:49:28.074460 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:49:28.077212 15332 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:49:28.080273 15332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:49:28.095108 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.089261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:40003 (local address 127.0.0.1:55422)
0923 14:49:28.090228 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:28.090279 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:28.090429 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:28.091270 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:28.091276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:28.091300 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:28.091400 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.091406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.092535 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.092540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.093715 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.093724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.094539 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.094542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.094656 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.094694 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 14:49:28.094836 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 14:49:28.094958 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":37,"spinlock_wait_cycles":11264,"thread_start_us":106,"threads_started":1}
I20260923 14:49:28.095114 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.089487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.30.2.195:40003)
0923 14:49:28.090803 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:28.090815 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:28.090837 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:49:28.090901 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:28.090906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:28.091078 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:28.091252 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:28.091527 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.092409 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.093838 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.094407 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.094527 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.094685 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:28.094872 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 14:49:28.094963 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"thread_start_us":134,"threads_started":1}
I20260923 14:49:28.095474 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.089733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:39765 (local address 127.0.0.1:40102)
0923 14:49:28.090267 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:28.090297 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:28.090429 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:28.091006 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:28.091013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:28.091037 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:28.091206 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.091236 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.092908 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.092914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.093830 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.093841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.094978 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.094982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.095100 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.095134 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 14:49:28.095185 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 14:49:28.095315 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":16,"spinlock_wait_cycles":3968,"thread_start_us":85,"threads_started":1}
I20260923 14:49:28.096818 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.090041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.30.2.193:39765)
0923 14:49:28.090450 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:28.090471 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:28.090500 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:49:28.090599 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:28.090607 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:28.090826 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:28.091019 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:28.091377 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.092725 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.093964 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.094826 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.096167 (+ 1341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.096398 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:28.096460 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 14:49:28.096633 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":92,"threads_started":1}
I20260923 14:49:28.096897 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.089988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:38821 (local address 127.0.0.1:42984)
0923 14:49:28.090665 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:28.090678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:28.090766 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:28.091394 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:28.091397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:28.091416 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:28.091522 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.091527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.092998 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.093001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.095020 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.095029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.096492 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.096496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.096586 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.096610 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 14:49:28.096688 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:49:28.096755 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260923 14:49:28.097380 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.090148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.30.2.194:38821)
0923 14:49:28.090718 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:28.090731 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:28.090752 (+ 21us) server_negotiation.cc:407] Connection header received
0923 14:49:28.090901 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:28.090906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:28.091078 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:28.091219 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:28.091669 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.092852 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.095166 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.096174 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.096296 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.096424 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:28.097113 (+ 689us) server_negotiation.cc:299] Negotiation successful
0923 14:49:28.097212 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":134,"threads_started":1}
I20260923 14:49:28.098773 15729 tablet_service.cc:1467] Processing CreateTablet for tablet 2d334312a378459f8c35feb6235fb5c3 (DEFAULT_TABLE table=test-table [id=47b7c83524024838825a2bb84ce958e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:28.099996 15729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d334312a378459f8c35feb6235fb5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:28.099610 15458 tablet_service.cc:1467] Processing CreateTablet for tablet 2d334312a378459f8c35feb6235fb5c3 (DEFAULT_TABLE table=test-table [id=47b7c83524024838825a2bb84ce958e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:28.099641 15593 tablet_service.cc:1467] Processing CreateTablet for tablet 2d334312a378459f8c35feb6235fb5c3 (DEFAULT_TABLE table=test-table [id=47b7c83524024838825a2bb84ce958e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:28.100665 15458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d334312a378459f8c35feb6235fb5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:28.100837 15593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d334312a378459f8c35feb6235fb5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:28.107035 15812 tablet_bootstrap.cc:492] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: Bootstrap starting.
I20260923 14:49:28.108330 15814 tablet_bootstrap.cc:492] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: Bootstrap starting.
I20260923 14:49:28.109326 15812 tablet_bootstrap.cc:654] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:28.110003 15814 tablet_bootstrap.cc:654] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:28.110498 15812 log.cc:824] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:28.111200 15814 log.cc:824] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:28.111304 15813 tablet_bootstrap.cc:492] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf: Bootstrap starting.
I20260923 14:49:28.112697 15812 tablet_bootstrap.cc:492] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: No bootstrap required, opened a new log
I20260923 14:49:28.112900 15812 ts_tablet_manager.cc:1397] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:49:28.113221 15814 tablet_bootstrap.cc:492] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: No bootstrap required, opened a new log
I20260923 14:49:28.113305 15813 tablet_bootstrap.cc:654] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:28.113421 15814 ts_tablet_manager.cc:1397] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:49:28.114347 15813 log.cc:824] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:28.116011 15813 tablet_bootstrap.cc:492] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf: No bootstrap required, opened a new log
I20260923 14:49:28.116210 15813 ts_tablet_manager.cc:1397] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:49:28.116873 15812 raft_consensus.cc:348] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16088807d03247a4afdb3a4017de907f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40003 } } peers { permanent_uuid: "2409558ed6b3420fb9910b01743ccc56" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 38821 } } peers { permanent_uuid: "aa175fd231e541f5a11d05cdc5633eaf" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39765 } }
I20260923 14:49:28.117149 15812 raft_consensus.cc:374] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:28.117203 15812 raft_consensus.cc:729] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2409558ed6b3420fb9910b01743ccc56, State: Initialized, Role: FOLLOWER
I20260923 14:49:28.117096 15814 raft_consensus.cc:348] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16088807d03247a4afdb3a4017de907f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40003 } } peers { permanent_uuid: "2409558ed6b3420fb9910b01743ccc56" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 38821 } } peers { permanent_uuid: "aa175fd231e541f5a11d05cdc5633eaf" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39765 } }
I20260923 14:49:28.117367 15814 raft_consensus.cc:374] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:28.117426 15814 raft_consensus.cc:729] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16088807d03247a4afdb3a4017de907f, State: Initialized, Role: FOLLOWER
I20260923 14:49:28.117875 15814 consensus_queue.cc:260] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16088807d03247a4afdb3a4017de907f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40003 } } peers { permanent_uuid: "2409558ed6b3420fb9910b01743ccc56" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 38821 } } peers { permanent_uuid: "aa175fd231e541f5a11d05cdc5633eaf" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39765 } }
I20260923 14:49:28.118286 15812 consensus_queue.cc:260] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16088807d03247a4afdb3a4017de907f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40003 } } peers { permanent_uuid: "2409558ed6b3420fb9910b01743ccc56" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 38821 } } peers { permanent_uuid: "aa175fd231e541f5a11d05cdc5633eaf" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39765 } }
I20260923 14:49:28.118850 15814 ts_tablet_manager.cc:1428] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 14:49:28.118983 15660 heartbeater.cc:503] Master 127.30.2.254:40041 was elected leader, sending a full tablet report...
I20260923 14:49:28.119148 15812 ts_tablet_manager.cc:1428] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 14:49:28.119171 15795 heartbeater.cc:503] Master 127.30.2.254:40041 was elected leader, sending a full tablet report...
I20260923 14:49:28.120997 15813 raft_consensus.cc:348] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16088807d03247a4afdb3a4017de907f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40003 } } peers { permanent_uuid: "2409558ed6b3420fb9910b01743ccc56" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 38821 } } peers { permanent_uuid: "aa175fd231e541f5a11d05cdc5633eaf" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39765 } }
I20260923 14:49:28.121356 15813 raft_consensus.cc:374] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:28.121423 15813 raft_consensus.cc:729] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa175fd231e541f5a11d05cdc5633eaf, State: Initialized, Role: FOLLOWER
I20260923 14:49:28.121874 15813 consensus_queue.cc:260] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16088807d03247a4afdb3a4017de907f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40003 } } peers { permanent_uuid: "2409558ed6b3420fb9910b01743ccc56" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 38821 } } peers { permanent_uuid: "aa175fd231e541f5a11d05cdc5633eaf" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39765 } }
I20260923 14:49:28.122534 15525 heartbeater.cc:503] Master 127.30.2.254:40041 was elected leader, sending a full tablet report...
I20260923 14:49:28.122766 15813 ts_tablet_manager.cc:1428] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:49:28.285795 15818 raft_consensus.cc:482] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:49:28.286022 15818 raft_consensus.cc:504] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16088807d03247a4afdb3a4017de907f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40003 } } peers { permanent_uuid: "2409558ed6b3420fb9910b01743ccc56" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 38821 } } peers { permanent_uuid: "aa175fd231e541f5a11d05cdc5633eaf" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39765 } }
I20260923 14:49:28.287269 15818 leader_election.cc:288] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2409558ed6b3420fb9910b01743ccc56 (127.30.2.194:38821), aa175fd231e541f5a11d05cdc5633eaf (127.30.2.193:39765)
I20260923 14:49:28.290964 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.287555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:38821 (local address 127.30.2.195:44535)
0923 14:49:28.287702 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:28.287720 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:28.287855 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:28.288142 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:28.288145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:28.288163 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:28.288304 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.288310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.289050 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.289055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.289843 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.289852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.290523 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.290527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.290632 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.290661 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 14:49:28.290744 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 14:49:28.290825 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"spinlock_wait_cycles":26368}
I20260923 14:49:28.290982 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.287677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:44535 (local address 127.30.2.194:38821)
0923 14:49:28.287826 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:28.287831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:28.287848 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:49:28.287988 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:28.287992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:28.288043 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:28.288132 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:28.288422 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.288929 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.289971 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.290395 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.290529 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.290583 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:28.290775 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 14:49:28.290881 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 14:49:28.291574 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.287555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:39765 (local address 127.30.2.195:37759)
0923 14:49:28.288061 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:28.288074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:28.288231 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:28.288519 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:28.288522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:28.288540 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:28.288675 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.288680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.289458 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.289463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.290433 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.290443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.291201 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.291207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.291314 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.291340 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 14:49:28.291419 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 14:49:28.291467 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":129,"spinlock_wait_cycles":768,"thread_start_us":196,"threads_started":1}
I20260923 14:49:28.291654 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.287659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:37759 (local address 127.30.2.193:39765)
0923 14:49:28.287830 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:28.287836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:28.288159 (+ 323us) server_negotiation.cc:407] Connection header received
0923 14:49:28.288356 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:28.288359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:28.288415 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:28.288508 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:28.288798 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.289336 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.290563 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.291006 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.291171 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.291223 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:28.291442 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 14:49:28.291522 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:49:28.291694 15614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d334312a378459f8c35feb6235fb5c3" candidate_uuid: "16088807d03247a4afdb3a4017de907f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2409558ed6b3420fb9910b01743ccc56" is_pre_election: true
I20260923 14:49:28.292075 15614 raft_consensus.cc:2454] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16088807d03247a4afdb3a4017de907f in term 0.
I20260923 14:49:28.292363 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d334312a378459f8c35feb6235fb5c3" candidate_uuid: "16088807d03247a4afdb3a4017de907f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa175fd231e541f5a11d05cdc5633eaf" is_pre_election: true
I20260923 14:49:28.292574 15682 leader_election.cc:302] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16088807d03247a4afdb3a4017de907f, 2409558ed6b3420fb9910b01743ccc56; no voters:
I20260923 14:49:28.292627 15479 raft_consensus.cc:2454] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16088807d03247a4afdb3a4017de907f in term 0.
I20260923 14:49:28.292896 15818 raft_consensus.cc:2781] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:49:28.293030 15818 raft_consensus.cc:482] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:49:28.293092 15818 raft_consensus.cc:3037] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:28.294111 15818 raft_consensus.cc:504] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16088807d03247a4afdb3a4017de907f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40003 } } peers { permanent_uuid: "2409558ed6b3420fb9910b01743ccc56" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 38821 } } peers { permanent_uuid: "aa175fd231e541f5a11d05cdc5633eaf" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39765 } }
I20260923 14:49:28.294451 15818 leader_election.cc:288] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [CANDIDATE]: Term 1 election: Requested vote from peers 2409558ed6b3420fb9910b01743ccc56 (127.30.2.194:38821), aa175fd231e541f5a11d05cdc5633eaf (127.30.2.193:39765)
I20260923 14:49:28.294834 15614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d334312a378459f8c35feb6235fb5c3" candidate_uuid: "16088807d03247a4afdb3a4017de907f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2409558ed6b3420fb9910b01743ccc56"
I20260923 14:49:28.294866 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d334312a378459f8c35feb6235fb5c3" candidate_uuid: "16088807d03247a4afdb3a4017de907f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa175fd231e541f5a11d05cdc5633eaf"
I20260923 14:49:28.294952 15614 raft_consensus.cc:3037] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:28.295042 15479 raft_consensus.cc:3037] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf [term 0 FOLLOWER]: Advancing to term 1
W20260923 14:49:28.296137 15796 tablet.cc:2369] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:49:28.296160 15614 raft_consensus.cc:2454] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16088807d03247a4afdb3a4017de907f in term 1.
I20260923 14:49:28.296466 15479 raft_consensus.cc:2454] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16088807d03247a4afdb3a4017de907f in term 1.
I20260923 14:49:28.296844 15681 leader_election.cc:302] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16088807d03247a4afdb3a4017de907f, aa175fd231e541f5a11d05cdc5633eaf; no voters:
I20260923 14:49:28.297050 15818 raft_consensus.cc:2781] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:28.297279 15818 raft_consensus.cc:686] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 1 LEADER]: Becoming Leader. State: Replica: 16088807d03247a4afdb3a4017de907f, State: Running, Role: LEADER
I20260923 14:49:28.297585 15818 consensus_queue.cc:237] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16088807d03247a4afdb3a4017de907f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40003 } } peers { permanent_uuid: "2409558ed6b3420fb9910b01743ccc56" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 38821 } } peers { permanent_uuid: "aa175fd231e541f5a11d05cdc5633eaf" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39765 } }
I20260923 14:49:28.300877 15330 catalog_manager.cc:5495] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f reported cstate change: term changed from 0 to 1, leader changed from <none> to 16088807d03247a4afdb3a4017de907f (127.30.2.195). New cstate: current_term: 1 leader_uuid: "16088807d03247a4afdb3a4017de907f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16088807d03247a4afdb3a4017de907f" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 40003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2409558ed6b3420fb9910b01743ccc56" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 38821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa175fd231e541f5a11d05cdc5633eaf" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39765 } health_report { overall_health: UNKNOWN } } }
W20260923 14:49:28.322794 15661 tablet.cc:2369] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:49:28.330451 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.326466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.30.2.195:40003)
0923 14:49:28.326653 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:28.326660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:28.326674 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:49:28.326908 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:28.326911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:28.326971 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:28.327087 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:49:28.327362 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.327975 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.328916 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.329183 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.329304 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.329627 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:49:28.330025 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:49:28.330167 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:28.330244 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 14:49:28.330302 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 14:49:28.336654 15614 raft_consensus.cc:1261] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56 [term 1 FOLLOWER]: Refusing update from remote peer 16088807d03247a4afdb3a4017de907f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:28.336654 15479 raft_consensus.cc:1261] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf [term 1 FOLLOWER]: Refusing update from remote peer 16088807d03247a4afdb3a4017de907f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:28.337487 15818 consensus_queue.cc:1035] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa175fd231e541f5a11d05cdc5633eaf" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 39765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:28.337743 15822 consensus_queue.cc:1035] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2409558ed6b3420fb9910b01743ccc56" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:28.346201 15826 mvcc.cc:205] Tried to move back new op lower bound from 7332556670299623424 to 7332556670149320704. Current Snapshot: MvccSnapshot[applied={T|T < 7332556670299623424}]
I20260923 14:49:28.346876 15827 mvcc.cc:205] Tried to move back new op lower bound from 7332556670299623424 to 7332556670149320704. Current Snapshot: MvccSnapshot[applied={T|T < 7332556670299623424}]
I20260923 14:49:28.354489 15330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:28.354655 15330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:28.357174 15330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abd792ae3c24485fa4b2810b6c9f0f5f: Sending DeleteTablet for 3 replicas of tablet 2d334312a378459f8c35feb6235fb5c3
I20260923 14:49:28.358038 15458 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d334312a378459f8c35feb6235fb5c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:28 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:40102
I20260923 14:49:28.358146 15729 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d334312a378459f8c35feb6235fb5c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:28 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:55422
I20260923 14:49:28.358269 15593 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d334312a378459f8c35feb6235fb5c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:28 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:42984
I20260923 14:49:28.358729 15837 tablet_replica.cc:330] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: stopping tablet replica
I20260923 14:49:28.358719 15836 tablet_replica.cc:330] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf: stopping tablet replica
I20260923 14:49:28.359015 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 15394
I20260923 14:49:28.359072 15837 raft_consensus.cc:2229] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:49:28.359128 15836 raft_consensus.cc:2229] T 2d334312a378459f8c35feb6235fb5c3 P aa175fd231e541f5a11d05cdc5633eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:49:28.359477 15837 raft_consensus.cc:2258] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:28.359977 15838 tablet_replica.cc:330] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: stopping tablet replica
I20260923 14:49:28.360258 15838 raft_consensus.cc:2229] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:49:28.360540 15838 raft_consensus.cc:2258] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:28.362077 15838 ts_tablet_manager.cc:1905] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:49:28.362077 15837 ts_tablet_manager.cc:1905] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:49:28.365011 15838 ts_tablet_manager.cc:1918] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:49:28.365011 15837 ts_tablet_manager.cc:1918] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:49:28.365113 15837 log.cc:1196] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/2d334312a378459f8c35feb6235fb5c3
I20260923 14:49:28.365232 15838 log.cc:1196] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/2d334312a378459f8c35feb6235fb5c3
I20260923 14:49:28.365486 15837 ts_tablet_manager.cc:1939] T 2d334312a378459f8c35feb6235fb5c3 P 16088807d03247a4afdb3a4017de907f: Deleting consensus metadata
I20260923 14:49:28.365754 15838 ts_tablet_manager.cc:1939] T 2d334312a378459f8c35feb6235fb5c3 P 2409558ed6b3420fb9910b01743ccc56: Deleting consensus metadata
I20260923 14:49:28.366791 15317 catalog_manager.cc:4841] TS 16088807d03247a4afdb3a4017de907f (127.30.2.195:40003): tablet 2d334312a378459f8c35feb6235fb5c3 (table test-table [id=47b7c83524024838825a2bb84ce958e3]) successfully deleted
I20260923 14:49:28.367144 15317 catalog_manager.cc:4841] TS 2409558ed6b3420fb9910b01743ccc56 (127.30.2.194:38821): tablet 2d334312a378459f8c35feb6235fb5c3 (table test-table [id=47b7c83524024838825a2bb84ce958e3]) successfully deleted
W20260923 14:49:28.369879 15316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:39765 (error 108)
I20260923 14:49:28.370087 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 15530
I20260923 14:49:28.371444 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.371107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:39765 (local address 127.0.0.1:40114)
0923 14:49:28.371311 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:28.371373 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260923 14:49:28.371644 15316 catalog_manager.cc:4584] TS aa175fd231e541f5a11d05cdc5633eaf (127.30.2.193:39765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d334312a378459f8c35feb6235fb5c3: Network error: Client connection negotiation failed: client connection to 127.30.2.193:39765: connect: Connection refused (error 111)
I20260923 14:49:28.379653 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 15665
I20260923 14:49:28.384873 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 15299
2026-09-23T14:49:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2463 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T14:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:28Z Disabled control of system clock
I20260923 14:49:28.417042 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:35857
--webserver_interface=127.30.2.254
--webserver_port=0
--builtin_ntp_servers=127.30.2.212:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:35857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:49:28.527417 15843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:28.527657 15843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:28.527691 15843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:28.531628 15843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:28.531679 15843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:28.531695 15843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:28.531711 15843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:28.531728 15843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:28.536540 15843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:35857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:35857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:28.537554 15843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:28.538915 15843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:28.547309 15851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:28.547367 15849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:28.547387 15848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:28.547976 15843 server_base.cc:1034] running on GCE node
I20260923 14:49:28.548556 15843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:28.549708 15843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:28.550923 15843 hybrid_clock.cc:648] HybridClock initialized: now 1790174968550906 us; error 38 us; skew 500 ppm
I20260923 14:49:28.552794 15843 webserver.cc:460] Webserver started at http://127.30.2.254:46229/ using document root <none> and password file <none>
I20260923 14:49:28.553393 15843 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:28.553511 15843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:28.553777 15843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:28.555904 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "a058004b60424422b13655dbfa470cf1"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "8ff7296f253363b291d3944fb8c8abee"
server_key_iv: "86a0d1db6fd769b6f43ec6ec9e657c7e"
server_key_version: "encryptionkey@0"
I20260923 14:49:28.556476 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "a058004b60424422b13655dbfa470cf1"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "8ff7296f253363b291d3944fb8c8abee"
server_key_iv: "86a0d1db6fd769b6f43ec6ec9e657c7e"
server_key_version: "encryptionkey@0"
I20260923 14:49:28.560292 15843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 14:49:28.562767 15857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:28.564194 15843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:49:28.564340 15843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "a058004b60424422b13655dbfa470cf1"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "8ff7296f253363b291d3944fb8c8abee"
server_key_iv: "86a0d1db6fd769b6f43ec6ec9e657c7e"
server_key_version: "encryptionkey@0"
I20260923 14:49:28.564453 15843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:28.582100 15843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:28.582770 15843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:28.582981 15843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:28.590230 15909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:35857 every 8 connection(s)
I20260923 14:49:28.590231 15843 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:35857
I20260923 14:49:28.591209 15843 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:28.592357 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 15843
I20260923 14:49:28.592484 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:28.592803 30731 external_mini_cluster.cc:1428] Setting key a5dd03450f194998bbf9be6592e281c4
I20260923 14:49:28.595602 15910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:28.601158 15910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1: Bootstrap starting.
I20260923 14:49:28.601709 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.594122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.30.2.254:35857)
0923 14:49:28.594729 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:28.594744 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:28.594779 (+ 35us) server_negotiation.cc:407] Connection header received
0923 14:49:28.595419 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:28.595434 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:28.595915 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:28.596232 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:28.596657 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.597584 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.598481 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.598941 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.599119 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.600035 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:28.600039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:28.600050 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:28.600415 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:28.600467 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:28.600487 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:28.600593 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:28.600748 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 14:49:28.600980 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
I20260923 14:49:28.603585 15910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:28.604422 15910 log.cc:824] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:28.606063 15910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1: No bootstrap required, opened a new log
I20260923 14:49:28.609543 15910 raft_consensus.cc:348] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a058004b60424422b13655dbfa470cf1" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35857 } }
I20260923 14:49:28.609802 15910 raft_consensus.cc:374] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:28.609886 15910 raft_consensus.cc:729] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a058004b60424422b13655dbfa470cf1, State: Initialized, Role: FOLLOWER
I20260923 14:49:28.610327 15910 consensus_queue.cc:260] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a058004b60424422b13655dbfa470cf1" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35857 } }
I20260923 14:49:28.610443 15910 raft_consensus.cc:388] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:28.610487 15910 raft_consensus.cc:482] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:28.610586 15910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:28.611337 15910 raft_consensus.cc:504] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a058004b60424422b13655dbfa470cf1" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35857 } }
I20260923 14:49:28.611613 15910 leader_election.cc:302] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a058004b60424422b13655dbfa470cf1; no voters:
I20260923 14:49:28.611915 15910 leader_election.cc:288] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:28.612030 15915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:28.612337 15915 raft_consensus.cc:686] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [term 1 LEADER]: Becoming Leader. State: Replica: a058004b60424422b13655dbfa470cf1, State: Running, Role: LEADER
I20260923 14:49:28.612622 15915 consensus_queue.cc:237] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a058004b60424422b13655dbfa470cf1" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35857 } }
I20260923 14:49:28.613071 15910 sys_catalog.cc:564] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:28.614269 15917 sys_catalog.cc:455] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a058004b60424422b13655dbfa470cf1. Latest consensus state: current_term: 1 leader_uuid: "a058004b60424422b13655dbfa470cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a058004b60424422b13655dbfa470cf1" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35857 } } }
I20260923 14:49:28.614406 15917 sys_catalog.cc:458] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:28.614677 15916 sys_catalog.cc:455] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a058004b60424422b13655dbfa470cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a058004b60424422b13655dbfa470cf1" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 35857 } } }
I20260923 14:49:28.614781 15916 sys_catalog.cc:458] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:28.614876 15924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:28.618306 15924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:28.623651 15924 catalog_manager.cc:1347] Generated new cluster ID: f3f0574306734c2e8f12b565a3890a03
I20260923 14:49:28.623718 15924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:28.652436 15924 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:28.653724 15924 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:28.659569 15924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1: Generated new TSK 0
I20260923 14:49:28.660257 15924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:28.664800 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35857
--builtin_ntp_servers=127.30.2.212:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:49:28.777334 15934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:28.777565 15934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:28.777601 15934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:28.781658 15934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:28.781708 15934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:28.781759 15934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:49:28.786530 15934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:28.787568 15934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:28.788998 15934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:28.798461 15942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:28.798442 15940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:28.798499 15934 server_base.cc:1034] running on GCE node
W20260923 14:49:28.798434 15939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:28.799008 15934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:28.799605 15934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:28.800796 15934 hybrid_clock.cc:648] HybridClock initialized: now 1790174968800787 us; error 53 us; skew 500 ppm
I20260923 14:49:28.802563 15934 webserver.cc:460] Webserver started at http://127.30.2.193:45719/ using document root <none> and password file <none>
I20260923 14:49:28.803125 15934 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:28.803215 15934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:28.803452 15934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:28.805459 15934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "f05e0bc34a801bd4b9eeab5e7f9bf284"
server_key_iv: "c3fccef96a93829894e3e53c4ed91acd"
server_key_version: "encryptionkey@0"
I20260923 14:49:28.806005 15934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "f05e0bc34a801bd4b9eeab5e7f9bf284"
server_key_iv: "c3fccef96a93829894e3e53c4ed91acd"
server_key_version: "encryptionkey@0"
I20260923 14:49:28.809540 15934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:49:28.811939 15948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:28.813079 15934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:28.813212 15934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "f05e0bc34a801bd4b9eeab5e7f9bf284"
server_key_iv: "c3fccef96a93829894e3e53c4ed91acd"
server_key_version: "encryptionkey@0"
I20260923 14:49:28.813329 15934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:28.844484 15934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:28.845157 15934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:28.845356 15934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:28.845942 15934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:28.846896 15934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:28.846968 15934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:28.847039 15934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:28.847088 15934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:28.855644 15934 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:37877
I20260923 14:49:28.855676 16061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:37877 every 8 connection(s)
I20260923 14:49:28.856525 15934 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:49:28.861367 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 15934
I20260923 14:49:28.861500 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:49:28.861864 30731 external_mini_cluster.cc:1428] Setting key da7421e960aa31fe93c4817455b1d8ae
I20260923 14:49:28.864077 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35857
--builtin_ntp_servers=127.30.2.212:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 14:49:28.866813 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.858262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:45347 (local address 127.30.2.254:35857)
0923 14:49:28.858412 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:28.858416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:28.859362 (+ 946us) server_negotiation.cc:407] Connection header received
0923 14:49:28.860808 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:28.860811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:28.860862 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:28.860941 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:28.862121 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.862823 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.864531 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.864748 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.864831 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.866159 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:28.866166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:28.866170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:28.866188 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:28.866215 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:28.866227 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:28.866311 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:28.866549 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 14:49:28.866648 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 14:49:28.867707 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:28.858617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35857 (local address 127.30.2.193:45347)
0923 14:49:28.859238 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:28.859283 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:28.860623 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:28.861134 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:28.861144 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:28.861252 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:28.861969 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.861992 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.862968 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.862970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.864407 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:28.864423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:28.864907 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:28.864913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:28.865163 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:28.865899 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:49:28.865900 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:28.865940 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:49:28.866003 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:28.866380 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:28.866389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:28.866395 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 14:49:28.866531 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 14:49:28.866791 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260923 14:49:28.869027 16062 heartbeater.cc:347] Connected to a master server at 127.30.2.254:35857
I20260923 14:49:28.869258 16062 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:28.869767 16062 heartbeater.cc:511] Master 127.30.2.254:35857 requested a full tablet report, sending...
I20260923 14:49:28.871256 15874 ts_manager.cc:194] Registered new tserver with Master: b9a1d5cd629b4c2eae5a6f2fdff6ea06 (127.30.2.193:37877)
I20260923 14:49:28.872445 15874 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.193:45347
W20260923 14:49:28.974674 16065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:28.974900 16065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:28.974933 16065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:28.978895 16065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:28.978945 16065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:28.978995 16065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:49:28.983676 16065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:28.984663 16065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:28.985982 16065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:28.995441 16071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:28.995628 16065 server_base.cc:1034] running on GCE node
W20260923 14:49:28.995402 16070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:28.995421 16073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:28.996150 16065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:28.996747 16065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:28.997905 16065 hybrid_clock.cc:648] HybridClock initialized: now 1790174968997888 us; error 27 us; skew 500 ppm
I20260923 14:49:28.999747 16065 webserver.cc:460] Webserver started at http://127.30.2.194:42789/ using document root <none> and password file <none>
I20260923 14:49:29.000366 16065 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:29.000427 16065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:29.000588 16065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:29.002549 16065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "45d58748e89543febae756c81736426e"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "75ba64fb748518dbddf05df1ee12f1b0"
server_key_iv: "079bf04c727d25c128c77fd78e217b8a"
server_key_version: "encryptionkey@0"
I20260923 14:49:29.003057 16065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "45d58748e89543febae756c81736426e"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "75ba64fb748518dbddf05df1ee12f1b0"
server_key_iv: "079bf04c727d25c128c77fd78e217b8a"
server_key_version: "encryptionkey@0"
I20260923 14:49:29.006590 16065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260923 14:49:29.008913 16079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:29.010090 16065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260923 14:49:29.010228 16065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "45d58748e89543febae756c81736426e"
format_stamp: "Formatted at 2026-09-23 14:49:28 on dist-test-slave-wjf2"
server_key: "75ba64fb748518dbddf05df1ee12f1b0"
server_key_iv: "079bf04c727d25c128c77fd78e217b8a"
server_key_version: "encryptionkey@0"
I20260923 14:49:29.010339 16065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:29.037731 16065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:29.038451 16065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:29.038652 16065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:29.039268 16065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:29.040258 16065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:29.040330 16065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:29.040402 16065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:29.040454 16065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:29.049301 16065 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:37851
I20260923 14:49:29.049325 16192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:37851 every 8 connection(s)
I20260923 14:49:29.050172 16065 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:49:29.051620 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 16065
I20260923 14:49:29.051748 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:49:29.052137 30731 external_mini_cluster.cc:1428] Setting key 5f904ed15eaf32f1f7da77dbc438db9a
I20260923 14:49:29.054273 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35857
--builtin_ntp_servers=127.30.2.212:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 14:49:29.060256 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.052070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:51347 (local address 127.30.2.254:35857)
0923 14:49:29.052216 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:29.052221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:29.053692 (+ 1471us) server_negotiation.cc:407] Connection header received
0923 14:49:29.055210 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:29.055217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:29.055281 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:29.055373 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:29.056575 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.057079 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.058272 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.058491 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.058600 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.059696 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:29.059700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:29.059702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:29.059721 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:29.059741 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:29.059748 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:29.059882 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:29.060087 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 14:49:29.060144 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:49:29.061525 16195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.052701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35857 (local address 127.30.2.194:51347)
0923 14:49:29.053543 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:29.053597 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:29.055000 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:29.055579 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:29.055590 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:29.055710 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:29.056433 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.056452 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.057192 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.057195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.058130 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.058139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.058638 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.058641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.058815 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.059434 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:49:29.059436 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:29.059476 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:49:29.059539 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:29.059897 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:29.059910 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:29.059920 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 14:49:29.060088 (+ 168us) client_negotiation.cc:240] Negotiation successful
0923 14:49:29.060414 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":150,"threads_started":1}
I20260923 14:49:29.062692 16193 heartbeater.cc:347] Connected to a master server at 127.30.2.254:35857
I20260923 14:49:29.062978 16193 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:29.063495 16193 heartbeater.cc:511] Master 127.30.2.254:35857 requested a full tablet report, sending...
I20260923 14:49:29.064601 15874 ts_manager.cc:194] Registered new tserver with Master: 45d58748e89543febae756c81736426e (127.30.2.194:37851)
I20260923 14:49:29.065156 15874 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.194:51347
W20260923 14:49:29.169113 16196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:29.169360 16196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:29.169397 16196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:29.173509 16196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:29.173557 16196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:29.173638 16196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:49:29.178339 16196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:29.179373 16196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:29.180891 16196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:29.190774 16201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:29.190845 16196 server_base.cc:1034] running on GCE node
W20260923 14:49:29.190778 16202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:29.190835 16204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:29.191550 16196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:29.192305 16196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:29.193511 16196 hybrid_clock.cc:648] HybridClock initialized: now 1790174969193497 us; error 27 us; skew 500 ppm
I20260923 14:49:29.195518 16196 webserver.cc:460] Webserver started at http://127.30.2.195:33597/ using document root <none> and password file <none>
I20260923 14:49:29.196250 16196 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:29.196373 16196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:29.196623 16196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:29.198828 16196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "b406cbef004b44e385b684dec6e63896"
format_stamp: "Formatted at 2026-09-23 14:49:29 on dist-test-slave-wjf2"
server_key: "43adfba10a509310098b0358e7605b85"
server_key_iv: "a93d4c4bdd7797d257522dfe748182bf"
server_key_version: "encryptionkey@0"
I20260923 14:49:29.199426 16196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "b406cbef004b44e385b684dec6e63896"
format_stamp: "Formatted at 2026-09-23 14:49:29 on dist-test-slave-wjf2"
server_key: "43adfba10a509310098b0358e7605b85"
server_key_iv: "a93d4c4bdd7797d257522dfe748182bf"
server_key_version: "encryptionkey@0"
I20260923 14:49:29.203358 16196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 14:49:29.205895 16210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:29.207327 16196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:49:29.207469 16196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "b406cbef004b44e385b684dec6e63896"
format_stamp: "Formatted at 2026-09-23 14:49:29 on dist-test-slave-wjf2"
server_key: "43adfba10a509310098b0358e7605b85"
server_key_iv: "a93d4c4bdd7797d257522dfe748182bf"
server_key_version: "encryptionkey@0"
I20260923 14:49:29.207585 16196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:29.226974 16196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:29.227708 16196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:29.227942 16196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:29.228571 16196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:29.229593 16196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:29.229669 16196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:29.229749 16196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:29.229782 16196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:29.238884 16196 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:34713
I20260923 14:49:29.238952 16324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:34713 every 8 connection(s)
I20260923 14:49:29.240263 16196 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:49:29.241437 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 16196
I20260923 14:49:29.241547 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:49:29.241900 30731 external_mini_cluster.cc:1428] Setting key 6987d18b207ab93a23a12972cd4a71af
I20260923 14:49:29.250195 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.242814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:36143 (local address 127.30.2.254:35857)
0923 14:49:29.242972 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:29.242977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:29.244201 (+ 1224us) server_negotiation.cc:407] Connection header received
0923 14:49:29.245901 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:29.245903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:29.245957 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:29.246025 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:29.246897 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.247406 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.248563 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.248771 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.248848 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.249804 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:29.249807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:29.249809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:29.249820 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:29.249839 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:29.249846 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:29.249902 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:29.250039 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 14:49:29.250087 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 14:49:29.250982 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.243161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:35857 (local address 127.30.2.195:36143)
0923 14:49:29.244051 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:29.244110 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:29.245702 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:29.246185 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:29.246192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:29.246273 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:29.246776 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.246788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.247509 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.247512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.248457 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.248467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.248896 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.248899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.249031 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.249571 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:49:29.249572 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:29.249610 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:49:29.249670 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:29.249918 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:29.249924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:29.249928 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 14:49:29.250017 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 14:49:29.250220 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":163,"threads_started":1}
I20260923 14:49:29.252240 16325 heartbeater.cc:347] Connected to a master server at 127.30.2.254:35857
I20260923 14:49:29.252486 16325 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:29.252997 16325 heartbeater.cc:511] Master 127.30.2.254:35857 requested a full tablet report, sending...
I20260923 14:49:29.254093 15874 ts_manager.cc:194] Registered new tserver with Master: b406cbef004b44e385b684dec6e63896 (127.30.2.195:34713)
I20260923 14:49:29.254642 15874 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.195:36143
I20260923 14:49:29.256654 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:49:29.262239 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.258527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.30.2.254:35857)
0923 14:49:29.258675 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:29.258680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:29.258917 (+ 237us) server_negotiation.cc:407] Connection header received
0923 14:49:29.258996 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:29.258998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:29.259037 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:29.259106 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:29.259318 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.259931 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.260741 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.260996 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.261079 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.261826 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:29.261829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:29.261831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:29.261843 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:29.261862 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:29.261870 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:29.261933 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:29.262008 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 14:49:29.262051 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:49:29.265012 15874 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:49:29.268066 15874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:49:29.282766 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.276570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.30.2.195:34713)
0923 14:49:29.277923 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:29.277933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:29.277953 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:49:29.278016 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:29.278020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:29.278173 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:29.278372 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:29.278677 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.279862 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.281580 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.282119 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.282247 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.282456 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:29.282515 (+ 59us) server_negotiation.cc:299] Negotiation successful
0923 14:49:29.282605 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":147,"threads_started":1}
I20260923 14:49:29.282764 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.276396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:34713 (local address 127.0.0.1:55376)
0923 14:49:29.276912 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:29.276978 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:29.277511 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:29.278363 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:29.278370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:29.278409 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:29.278549 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.278557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.279995 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.279999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.281435 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.281446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.282248 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.282251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.282354 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.282391 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 14:49:29.282493 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 14:49:29.282607 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":53,"spinlock_wait_cycles":2560,"thread_start_us":82,"threads_started":1}
I20260923 14:49:29.283358 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.276598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:37851 (local address 127.0.0.1:52894)
0923 14:49:29.277753 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:29.277763 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:29.277881 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:29.278540 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:29.278543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:29.278559 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:29.278669 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.278675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.279968 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.279972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.281220 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.281230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.282531 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.282538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.282639 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.282664 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 14:49:29.282739 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 14:49:29.282814 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":61,"thread_start_us":109,"threads_started":1}
I20260923 14:49:29.283710 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.277134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.30.2.194:37851)
0923 14:49:29.277484 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:29.277496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:29.277855 (+ 359us) server_negotiation.cc:407] Connection header received
0923 14:49:29.277996 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:29.278003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:29.278212 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:29.278382 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:29.278823 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.279823 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.281355 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.282137 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.282249 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.282456 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:29.283000 (+ 544us) server_negotiation.cc:299] Negotiation successful
0923 14:49:29.283543 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":139,"threads_started":1}
I20260923 14:49:29.284283 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.277444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.30.2.193:37877)
0923 14:49:29.278141 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:29.278152 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:29.278510 (+ 358us) server_negotiation.cc:407] Connection header received
0923 14:49:29.278591 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:29.278594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:29.278783 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:29.278942 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:29.279176 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.280256 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.281649 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.282491 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.282599 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.282790 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:29.283983 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0923 14:49:29.284108 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":105,"threads_started":1}
I20260923 14:49:29.284541 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.277110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:37877 (local address 127.0.0.1:35332)
0923 14:49:29.278308 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:29.278322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:29.278416 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:29.278941 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:29.278944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:29.278965 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:29.279053 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.279060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.280381 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.280384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.281522 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.281531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.283211 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.283217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.283313 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.283337 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 14:49:29.283608 (+ 271us) client_negotiation.cc:240] Negotiation successful
0923 14:49:29.283696 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"mutex_wait_us":42,"spinlock_wait_cycles":6144,"thread_start_us":94,"threads_started":1}
I20260923 14:49:29.285738 16259 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae9e5e456c2465f84c48d8d4f028d97 (DEFAULT_TABLE table=test-table [id=1439978dc5384b23b248ed26d3340bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:29.286728 16259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae9e5e456c2465f84c48d8d4f028d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:29.286758 16127 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae9e5e456c2465f84c48d8d4f028d97 (DEFAULT_TABLE table=test-table [id=1439978dc5384b23b248ed26d3340bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:29.287284 15996 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae9e5e456c2465f84c48d8d4f028d97 (DEFAULT_TABLE table=test-table [id=1439978dc5384b23b248ed26d3340bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:29.287894 16127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae9e5e456c2465f84c48d8d4f028d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:29.288319 15996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae9e5e456c2465f84c48d8d4f028d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:29.292990 16342 tablet_bootstrap.cc:492] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896: Bootstrap starting.
I20260923 14:49:29.295113 16342 tablet_bootstrap.cc:654] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:29.296147 16345 tablet_bootstrap.cc:492] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06: Bootstrap starting.
I20260923 14:49:29.296344 16344 tablet_bootstrap.cc:492] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e: Bootstrap starting.
I20260923 14:49:29.296408 16342 log.cc:824] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:29.298286 16345 tablet_bootstrap.cc:654] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:29.298329 16344 tablet_bootstrap.cc:654] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:29.298660 16342 tablet_bootstrap.cc:492] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896: No bootstrap required, opened a new log
I20260923 14:49:29.298848 16342 ts_tablet_manager.cc:1397] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:49:29.299108 16345 log.cc:824] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:29.299134 16344 log.cc:824] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:29.301146 16345 tablet_bootstrap.cc:492] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06: No bootstrap required, opened a new log
I20260923 14:49:29.301332 16344 tablet_bootstrap.cc:492] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e: No bootstrap required, opened a new log
I20260923 14:49:29.301359 16345 ts_tablet_manager.cc:1397] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:49:29.301522 16344 ts_tablet_manager.cc:1397] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:49:29.303316 16342 raft_consensus.cc:348] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b406cbef004b44e385b684dec6e63896" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 34713 } } peers { permanent_uuid: "45d58748e89543febae756c81736426e" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37851 } } peers { permanent_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37877 } }
I20260923 14:49:29.303601 16342 raft_consensus.cc:374] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:29.303663 16342 raft_consensus.cc:729] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b406cbef004b44e385b684dec6e63896, State: Initialized, Role: FOLLOWER
I20260923 14:49:29.304168 16342 consensus_queue.cc:260] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b406cbef004b44e385b684dec6e63896" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 34713 } } peers { permanent_uuid: "45d58748e89543febae756c81736426e" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37851 } } peers { permanent_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37877 } }
I20260923 14:49:29.304872 16325 heartbeater.cc:503] Master 127.30.2.254:35857 was elected leader, sending a full tablet report...
I20260923 14:49:29.305198 16342 ts_tablet_manager.cc:1428] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 14:49:29.305596 16344 raft_consensus.cc:348] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b406cbef004b44e385b684dec6e63896" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 34713 } } peers { permanent_uuid: "45d58748e89543febae756c81736426e" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37851 } } peers { permanent_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37877 } }
I20260923 14:49:29.305822 16344 raft_consensus.cc:374] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:29.305887 16344 raft_consensus.cc:729] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d58748e89543febae756c81736426e, State: Initialized, Role: FOLLOWER
I20260923 14:49:29.306283 16344 consensus_queue.cc:260] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b406cbef004b44e385b684dec6e63896" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 34713 } } peers { permanent_uuid: "45d58748e89543febae756c81736426e" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37851 } } peers { permanent_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37877 } }
I20260923 14:49:29.306304 16345 raft_consensus.cc:348] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b406cbef004b44e385b684dec6e63896" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 34713 } } peers { permanent_uuid: "45d58748e89543febae756c81736426e" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37851 } } peers { permanent_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37877 } }
I20260923 14:49:29.306586 16345 raft_consensus.cc:374] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:29.306636 16345 raft_consensus.cc:729] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a1d5cd629b4c2eae5a6f2fdff6ea06, State: Initialized, Role: FOLLOWER
I20260923 14:49:29.307120 16344 ts_tablet_manager.cc:1428] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 14:49:29.307046 16345 consensus_queue.cc:260] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b406cbef004b44e385b684dec6e63896" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 34713 } } peers { permanent_uuid: "45d58748e89543febae756c81736426e" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37851 } } peers { permanent_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37877 } }
I20260923 14:49:29.307430 16193 heartbeater.cc:503] Master 127.30.2.254:35857 was elected leader, sending a full tablet report...
I20260923 14:49:29.308053 16345 ts_tablet_manager.cc:1428] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260923 14:49:29.309005 16062 heartbeater.cc:503] Master 127.30.2.254:35857 was elected leader, sending a full tablet report...
W20260923 14:49:29.357761 16063 tablet.cc:2369] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:49:29.444999 16349 raft_consensus.cc:482] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:49:29.445222 16349 raft_consensus.cc:504] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b406cbef004b44e385b684dec6e63896" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 34713 } } peers { permanent_uuid: "45d58748e89543febae756c81736426e" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37851 } } peers { permanent_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37877 } }
I20260923 14:49:29.446545 16349 leader_election.cc:288] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45d58748e89543febae756c81736426e (127.30.2.194:37851), b9a1d5cd629b4c2eae5a6f2fdff6ea06 (127.30.2.193:37877)
I20260923 14:49:29.452033 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.447022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:52547 (local address 127.30.2.193:37877)
0923 14:49:29.447180 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:29.447188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:29.447449 (+ 261us) server_negotiation.cc:407] Connection header received
0923 14:49:29.447556 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:29.447559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:29.447629 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:29.447730 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:29.448065 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.448869 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.450685 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.451307 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.451439 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.451491 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:29.451844 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 14:49:29.451910 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 14:49:29.452033 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.446972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:36337 (local address 127.30.2.194:37851)
0923 14:49:29.447107 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:29.447112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:29.447128 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:49:29.447281 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:29.447284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:29.447341 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:29.447469 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:29.447830 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.448760 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.450685 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.451313 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.451439 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.451491 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:29.451831 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 14:49:29.451906 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 14:49:29.452075 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.446730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:37851 (local address 127.30.2.195:36337)
0923 14:49:29.446996 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:29.447014 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:29.447116 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:29.447493 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:29.447498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:29.447519 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:29.447637 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.447642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.448959 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.448965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.450493 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.450503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.451439 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.451443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.451654 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.451713 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 14:49:29.451814 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 14:49:29.451925 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":284}
I20260923 14:49:29.452083 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.446821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:37877 (local address 127.30.2.195:52547)
0923 14:49:29.447335 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:29.447346 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:29.447455 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:29.447740 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:29.447745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:29.447772 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:29.447935 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.447941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.448995 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.448998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.450492 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:29.450503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.451439 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.451443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:29.451658 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.451710 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 14:49:29.451814 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 14:49:29.451924 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":139,"thread_start_us":165,"threads_started":1}
I20260923 14:49:29.452730 16147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae9e5e456c2465f84c48d8d4f028d97" candidate_uuid: "b406cbef004b44e385b684dec6e63896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d58748e89543febae756c81736426e" is_pre_election: true
I20260923 14:49:29.452751 16016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae9e5e456c2465f84c48d8d4f028d97" candidate_uuid: "b406cbef004b44e385b684dec6e63896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" is_pre_election: true
I20260923 14:49:29.453034 16147 raft_consensus.cc:2454] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b406cbef004b44e385b684dec6e63896 in term 0.
I20260923 14:49:29.453043 16016 raft_consensus.cc:2454] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b406cbef004b44e385b684dec6e63896 in term 0.
I20260923 14:49:29.453547 16211 leader_election.cc:302] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45d58748e89543febae756c81736426e, b406cbef004b44e385b684dec6e63896, b9a1d5cd629b4c2eae5a6f2fdff6ea06; no voters:
I20260923 14:49:29.453819 16349 raft_consensus.cc:2781] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:49:29.453890 16349 raft_consensus.cc:482] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:49:29.453944 16349 raft_consensus.cc:3037] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:29.454908 16349 raft_consensus.cc:504] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b406cbef004b44e385b684dec6e63896" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 34713 } } peers { permanent_uuid: "45d58748e89543febae756c81736426e" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37851 } } peers { permanent_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37877 } }
I20260923 14:49:29.455235 16349 leader_election.cc:288] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [CANDIDATE]: Term 1 election: Requested vote from peers 45d58748e89543febae756c81736426e (127.30.2.194:37851), b9a1d5cd629b4c2eae5a6f2fdff6ea06 (127.30.2.193:37877)
I20260923 14:49:29.455587 16016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae9e5e456c2465f84c48d8d4f028d97" candidate_uuid: "b406cbef004b44e385b684dec6e63896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06"
I20260923 14:49:29.455592 16147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae9e5e456c2465f84c48d8d4f028d97" candidate_uuid: "b406cbef004b44e385b684dec6e63896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d58748e89543febae756c81736426e"
I20260923 14:49:29.455681 16016 raft_consensus.cc:3037] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:29.455711 16147 raft_consensus.cc:3037] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:29.456723 16147 raft_consensus.cc:2454] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b406cbef004b44e385b684dec6e63896 in term 1.
I20260923 14:49:29.456768 16016 raft_consensus.cc:2454] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b406cbef004b44e385b684dec6e63896 in term 1.
I20260923 14:49:29.457041 16214 leader_election.cc:302] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d58748e89543febae756c81736426e, b406cbef004b44e385b684dec6e63896; no voters:
I20260923 14:49:29.457269 16349 raft_consensus.cc:2781] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:29.457502 16349 raft_consensus.cc:686] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 1 LEADER]: Becoming Leader. State: Replica: b406cbef004b44e385b684dec6e63896, State: Running, Role: LEADER
I20260923 14:49:29.457854 16349 consensus_queue.cc:237] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b406cbef004b44e385b684dec6e63896" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 34713 } } peers { permanent_uuid: "45d58748e89543febae756c81736426e" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37851 } } peers { permanent_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37877 } }
I20260923 14:49:29.461012 15874 catalog_manager.cc:5495] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 reported cstate change: term changed from 0 to 1, leader changed from <none> to b406cbef004b44e385b684dec6e63896 (127.30.2.195). New cstate: current_term: 1 leader_uuid: "b406cbef004b44e385b684dec6e63896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b406cbef004b44e385b684dec6e63896" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 34713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45d58748e89543febae756c81736426e" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37877 } health_report { overall_health: UNKNOWN } } }
I20260923 14:49:29.473276 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.469819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.30.2.195:34713)
0923 14:49:29.469950 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:29.469954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:29.469969 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:49:29.470130 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:29.470132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:29.470181 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:29.470262 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:49:29.470481 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.470967 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.471962 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:29.472166 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:29.472244 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:29.472527 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:49:29.472919 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:49:29.473004 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:29.473065 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 14:49:29.473105 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:49:29.479009 16147 raft_consensus.cc:1261] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e [term 1 FOLLOWER]: Refusing update from remote peer b406cbef004b44e385b684dec6e63896: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:29.479696 16349 consensus_queue.cc:1035] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d58748e89543febae756c81736426e" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:29.479925 16016 raft_consensus.cc:1261] T 3ae9e5e456c2465f84c48d8d4f028d97 P b9a1d5cd629b4c2eae5a6f2fdff6ea06 [term 1 FOLLOWER]: Refusing update from remote peer b406cbef004b44e385b684dec6e63896: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:29.480496 16349 consensus_queue.cc:1035] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9a1d5cd629b4c2eae5a6f2fdff6ea06" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:29.488968 16358 mvcc.cc:205] Tried to move back new op lower bound from 7332556674978930688 to 7332556674902175744. Current Snapshot: MvccSnapshot[applied={T|T < 7332556674978930688}]
I20260923 14:49:29.490327 16357 mvcc.cc:205] Tried to move back new op lower bound from 7332556674978930688 to 7332556674902175744. Current Snapshot: MvccSnapshot[applied={T|T < 7332556674978930688 or (T in {7332556674978930688})}]
I20260923 14:49:29.495565 15874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:29.495728 15874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:29.498732 15874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a058004b60424422b13655dbfa470cf1: Sending DeleteTablet for 3 replicas of tablet 3ae9e5e456c2465f84c48d8d4f028d97
I20260923 14:49:29.499755 15996 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ae9e5e456c2465f84c48d8d4f028d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:29 UTC) from {username='slave'} at 127.0.0.1:35332
I20260923 14:49:29.499836 16259 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ae9e5e456c2465f84c48d8d4f028d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:29 UTC) from {username='slave'} at 127.0.0.1:55376
I20260923 14:49:29.500324 16367 tablet_replica.cc:330] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896: stopping tablet replica
I20260923 14:49:29.500455 16127 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ae9e5e456c2465f84c48d8d4f028d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:29 UTC) from {username='slave'} at 127.0.0.1:52894
I20260923 14:49:29.500793 16367 raft_consensus.cc:2229] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:49:29.500934 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 15934
I20260923 14:49:29.501183 16367 raft_consensus.cc:2258] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:29.502760 16367 ts_tablet_manager.cc:1905] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:49:29.505079 16367 ts_tablet_manager.cc:1918] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:49:29.505194 16367 log.cc:1196] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/3ae9e5e456c2465f84c48d8d4f028d97
I20260923 14:49:29.505508 16367 ts_tablet_manager.cc:1939] T 3ae9e5e456c2465f84c48d8d4f028d97 P b406cbef004b44e385b684dec6e63896: Deleting consensus metadata
I20260923 14:49:29.506561 15858 catalog_manager.cc:4841] TS b406cbef004b44e385b684dec6e63896 (127.30.2.195:34713): tablet 3ae9e5e456c2465f84c48d8d4f028d97 (table test-table [id=1439978dc5384b23b248ed26d3340bd2]) successfully deleted
I20260923 14:49:29.508005 16368 tablet_replica.cc:330] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e: stopping tablet replica
I20260923 14:49:29.508289 16368 raft_consensus.cc:2229] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:49:29.508586 16368 raft_consensus.cc:2258] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:29.509755 16368 ts_tablet_manager.cc:1905] T 3ae9e5e456c2465f84c48d8d4f028d97 P 45d58748e89543febae756c81736426e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 14:49:29.510506 15858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:37877 (error 108)
I20260923 14:49:29.511250 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 16065
I20260923 14:49:29.511600 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.511238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:37877 (local address 127.0.0.1:32878)
0923 14:49:29.511445 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:29.511525 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260923 14:49:29.511904 15858 catalog_manager.cc:4584] TS b9a1d5cd629b4c2eae5a6f2fdff6ea06 (127.30.2.193:37877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ae9e5e456c2465f84c48d8d4f028d97: Network error: Client connection negotiation failed: client connection to 127.30.2.193:37877: connect: Connection refused (error 111)
I20260923 14:49:29.520045 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 16196
I20260923 14:49:29.521696 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:29.520238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:37851 (local address 127.0.0.1:48984)
0923 14:49:29.521586 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:29.521632 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.194:37851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1238}
I20260923 14:49:29.525319 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 15843
2026-09-23T14:49:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1137 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:29 dist-test-slave-wjf2 krb5kdc[16372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:29 dist-test-slave-wjf2 krb5kdc[16372](info): set up 2 sockets
Sep 23 14:49:29 dist-test-slave-wjf2 krb5kdc[16372](info): commencing operation
krb5kdc: starting...
W20260923 14:49:30.569173 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:30 dist-test-slave-wjf2 krb5kdc[16372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:49:30.688072 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44799
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:44799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:30.798247 16385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:30.798496 16385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:30.798535 16385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:30.802518 16385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:30.802567 16385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:30.802585 16385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:30.802601 16385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:30.802615 16385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:30.807964 16385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:44799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:30.809072 16385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:30.810173 16385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:30.819531 16393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:30.819551 16390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:30.819552 16391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:30.819852 16385 server_base.cc:1034] running on GCE node
I20260923 14:49:30.820412 16385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:30.821657 16385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:30.822825 16385 hybrid_clock.cc:648] HybridClock initialized: now 1790174970822796 us; error 41 us; skew 500 ppm
Sep 23 14:49:30 dist-test-slave-wjf2 krb5kdc[16372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174970, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:30.825557 16385 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:49:30.826437 16385 webserver.cc:460] Webserver started at http://127.30.2.254:42735/ using document root <none> and password file <none>
I20260923 14:49:30.827050 16385 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:30.827142 16385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:30.827378 16385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:30.829473 16385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "aa9037d012194c37a05f11342cb3fa1e"
format_stamp: "Formatted at 2026-09-23 14:49:30 on dist-test-slave-wjf2"
server_key: "68adcb8748af723fd0d8d83bd41e6a9c"
server_key_iv: "cee189047aa3104d7e6c1830031c746c"
server_key_version: "encryptionkey@0"
I20260923 14:49:30.830056 16385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "aa9037d012194c37a05f11342cb3fa1e"
format_stamp: "Formatted at 2026-09-23 14:49:30 on dist-test-slave-wjf2"
server_key: "68adcb8748af723fd0d8d83bd41e6a9c"
server_key_iv: "cee189047aa3104d7e6c1830031c746c"
server_key_version: "encryptionkey@0"
I20260923 14:49:30.833810 16385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260923 14:49:30.836289 16400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:30.837500 16385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:30.837641 16385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "aa9037d012194c37a05f11342cb3fa1e"
format_stamp: "Formatted at 2026-09-23 14:49:30 on dist-test-slave-wjf2"
server_key: "68adcb8748af723fd0d8d83bd41e6a9c"
server_key_iv: "cee189047aa3104d7e6c1830031c746c"
server_key_version: "encryptionkey@0"
I20260923 14:49:30.837751 16385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:30.863070 16385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:30.866231 16385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:30.866438 16385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:30.874257 16385 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:44799
I20260923 14:49:30.874281 16452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:44799 every 8 connection(s)
I20260923 14:49:30.875264 16385 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:30.878391 16453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:30.884192 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 16385
I20260923 14:49:30.884292 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:30.884570 30731 external_mini_cluster.cc:1428] Setting key 4287e1ad62855815faf2f211fe3440b6
I20260923 14:49:30.884583 16453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e: Bootstrap starting.
I20260923 14:49:30.887707 16453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:30.888760 16453 log.cc:824] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:30.891194 16453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e: No bootstrap required, opened a new log
Sep 23 14:49:30 dist-test-slave-wjf2 krb5kdc[16372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:49:30.895637 16453 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa9037d012194c37a05f11342cb3fa1e" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44799 } }
I20260923 14:49:30.895949 16453 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:30.896018 16453 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa9037d012194c37a05f11342cb3fa1e, State: Initialized, Role: FOLLOWER
I20260923 14:49:30.896605 16453 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa9037d012194c37a05f11342cb3fa1e" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44799 } }
I20260923 14:49:30.896807 16453 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:30.896876 16453 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:30.896973 16453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:30.898121 16453 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa9037d012194c37a05f11342cb3fa1e" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44799 } }
I20260923 14:49:30.898481 16453 leader_election.cc:302] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa9037d012194c37a05f11342cb3fa1e; no voters:
I20260923 14:49:30.898818 16453 leader_election.cc:288] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:30.898888 16458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:30.899094 16458 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [term 1 LEADER]: Becoming Leader. State: Replica: aa9037d012194c37a05f11342cb3fa1e, State: Running, Role: LEADER
I20260923 14:49:30.899389 16458 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa9037d012194c37a05f11342cb3fa1e" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44799 } }
I20260923 14:49:30.900164 16453 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:30.901153 16460 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa9037d012194c37a05f11342cb3fa1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa9037d012194c37a05f11342cb3fa1e" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44799 } } }
I20260923 14:49:30.901433 16460 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:30.902115 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:30.886029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.30.2.254:44799)
0923 14:49:30.886520 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:30.886544 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:30.886593 (+ 49us) server_negotiation.cc:407] Connection header received
0923 14:49:30.887514 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:30.887541 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:30.888059 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:30.888488 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:30.889528 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:30.891004 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:30.892059 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:30.892744 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:30.892914 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:30.895709 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:30.895716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:30.895733 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:30.895766 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:30.898219 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:30.898687 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:30.898694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:30.898700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:30.898841 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:30.899238 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:30.899243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:30.899245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:30.899838 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:30.900064 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:30.900366 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 14:49:30.900699 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":119,"threads_started":1}
I20260923 14:49:30.901316 16459 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa9037d012194c37a05f11342cb3fa1e. Latest consensus state: current_term: 1 leader_uuid: "aa9037d012194c37a05f11342cb3fa1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa9037d012194c37a05f11342cb3fa1e" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44799 } } }
I20260923 14:49:30.902470 16459 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e [sys.catalog]: This master's current role is: LEADER
W20260923 14:49:30.905261 16473 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:49:30.905324 16473 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:49:30.905403 16467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:30.906195 16467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:30.911638 16467 catalog_manager.cc:1347] Generated new cluster ID: 2a503c0a972d4737b347a309b2ab09e4
I20260923 14:49:30.911697 16467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:30.922047 16467 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:30.922878 16467 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:30.926834 16467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e: Generated new TSK 0
I20260923 14:49:30.927428 16467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:49:30.967387 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44799
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:31.088033 16479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:31.088326 16479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:31.088414 16479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:31.092389 16479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:31.092463 16479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:31.092545 16479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:49:31.097656 16479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:31.098644 16479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:31.099740 16479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:31.109393 16484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:31.109442 16487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:31.109428 16485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:31.109632 16479 server_base.cc:1034] running on GCE node
I20260923 14:49:31.110023 16479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:31.110649 16479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:31.111863 16479 hybrid_clock.cc:648] HybridClock initialized: now 1790174971111845 us; error 25 us; skew 500 ppm
Sep 23 14:49:31 dist-test-slave-wjf2 krb5kdc[16372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:31.122203 16479 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:49:31.123097 16479 webserver.cc:460] Webserver started at http://127.30.2.193:45045/ using document root <none> and password file <none>
I20260923 14:49:31.123693 16479 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:31.123754 16479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:31.123983 16479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:31.125933 16479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "5081b5e5b2e2458198a0d64979004588"
format_stamp: "Formatted at 2026-09-23 14:49:31 on dist-test-slave-wjf2"
server_key: "eb023846dada6199d05ee5629d802ba2"
server_key_iv: "67ff152c9e80ccefe285cb1dbc9c2a8e"
server_key_version: "encryptionkey@0"
I20260923 14:49:31.126456 16479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "5081b5e5b2e2458198a0d64979004588"
format_stamp: "Formatted at 2026-09-23 14:49:31 on dist-test-slave-wjf2"
server_key: "eb023846dada6199d05ee5629d802ba2"
server_key_iv: "67ff152c9e80ccefe285cb1dbc9c2a8e"
server_key_version: "encryptionkey@0"
I20260923 14:49:31.130046 16479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:49:31.132360 16494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:31.133517 16479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:31.133631 16479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "5081b5e5b2e2458198a0d64979004588"
format_stamp: "Formatted at 2026-09-23 14:49:31 on dist-test-slave-wjf2"
server_key: "eb023846dada6199d05ee5629d802ba2"
server_key_iv: "67ff152c9e80ccefe285cb1dbc9c2a8e"
server_key_version: "encryptionkey@0"
I20260923 14:49:31.133733 16479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:31.147624 16479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:31.150493 16479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:31.150687 16479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:31.151258 16479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:31.152130 16479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:31.152179 16479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:31.152241 16479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:31.152309 16479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:31.161586 16479 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:37869
I20260923 14:49:31.161608 16607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:37869 every 8 connection(s)
I20260923 14:49:31.162454 16479 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:49:31.164376 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 16479
I20260923 14:49:31.164507 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:49:31.164816 30731 external_mini_cluster.cc:1428] Setting key c128126cf0f04bb3fa74cf48b7aa0188
Sep 23 14:49:31 dist-test-slave-wjf2 krb5kdc[16372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:49:31.176541 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.164375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:39865 (local address 127.30.2.254:44799)
0923 14:49:31.164560 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:31.164565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:31.165969 (+ 1404us) server_negotiation.cc:407] Connection header received
0923 14:49:31.166982 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:31.166988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:31.167048 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:31.167142 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:31.168337 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.168872 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.170014 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.170264 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.170387 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.172994 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:31.172997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:31.172999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:31.173022 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:31.174565 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:31.175023 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:31.175026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:31.175027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:31.175099 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:31.175519 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:31.175522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:31.175523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:31.175703 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:31.175829 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:31.176320 (+ 491us) server_negotiation.cc:299] Negotiation successful
0923 14:49:31.176425 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
I20260923 14:49:31.177541 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.164771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:44799 (local address 127.30.2.193:39865)
0923 14:49:31.165767 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:31.165861 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:31.166771 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:31.167301 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:31.167307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:31.167645 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:31.168188 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.168203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.169000 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.169005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.169876 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.169892 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.170424 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.170428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.170592 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.171200 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:49:31.171216 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:31.172804 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:31.174676 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:31.174681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:31.174692 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:31.174936 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:31.175207 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:31.175209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:31.175211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:31.175427 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:31.175866 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:31.175871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:31.176082 (+ 211us) client_negotiation.cc:769] Sending connection context
0923 14:49:31.176284 (+ 202us) client_negotiation.cc:240] Negotiation successful
0923 14:49:31.176521 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":614,"threads_started":1}
I20260923 14:49:31.178740 16608 heartbeater.cc:347] Connected to a master server at 127.30.2.254:44799
I20260923 14:49:31.178998 16608 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:31.179592 16608 heartbeater.cc:511] Master 127.30.2.254:44799 requested a full tablet report, sending...
I20260923 14:49:31.181138 16417 ts_manager.cc:194] Registered new tserver with Master: 5081b5e5b2e2458198a0d64979004588 (127.30.2.193:37869)
I20260923 14:49:31.182307 16417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:39865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:49:31.192600 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44799
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:31.304641 16613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:31.304883 16613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:31.304960 16613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:31.309028 16613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:31.309078 16613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:31.309165 16613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:49:31.314262 16613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:31.315305 16613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:31.316344 16613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:31.325984 16618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:31.326076 16613 server_base.cc:1034] running on GCE node
W20260923 14:49:31.326030 16619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:31.326004 16621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:31.326656 16613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:31.327303 16613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:31.328496 16613 hybrid_clock.cc:648] HybridClock initialized: now 1790174971328479 us; error 29 us; skew 500 ppm
Sep 23 14:49:31 dist-test-slave-wjf2 krb5kdc[16372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:31.331146 16613 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:49:31.332005 16613 webserver.cc:460] Webserver started at http://127.30.2.194:40295/ using document root <none> and password file <none>
I20260923 14:49:31.332543 16613 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:31.332631 16613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:31.332891 16613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:31.334844 16613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "3e284f77463349f98301e1ba4a775a5f"
format_stamp: "Formatted at 2026-09-23 14:49:31 on dist-test-slave-wjf2"
server_key: "373866916b1d12eb8fcb3cab2a2af451"
server_key_iv: "8ea857df793faf1364bb49b66502b174"
server_key_version: "encryptionkey@0"
I20260923 14:49:31.335374 16613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "3e284f77463349f98301e1ba4a775a5f"
format_stamp: "Formatted at 2026-09-23 14:49:31 on dist-test-slave-wjf2"
server_key: "373866916b1d12eb8fcb3cab2a2af451"
server_key_iv: "8ea857df793faf1364bb49b66502b174"
server_key_version: "encryptionkey@0"
I20260923 14:49:31.339534 16613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 14:49:31.342201 16628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:31.343531 16613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:49:31.343657 16613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "3e284f77463349f98301e1ba4a775a5f"
format_stamp: "Formatted at 2026-09-23 14:49:31 on dist-test-slave-wjf2"
server_key: "373866916b1d12eb8fcb3cab2a2af451"
server_key_iv: "8ea857df793faf1364bb49b66502b174"
server_key_version: "encryptionkey@0"
I20260923 14:49:31.343756 16613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:31.372448 16613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:31.375733 16613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:31.375990 16613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:31.376587 16613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:31.377401 16613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:31.377446 16613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:31.377511 16613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:31.377548 16613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:31.386500 16613 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:37651
I20260923 14:49:31.386526 16741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:37651 every 8 connection(s)
I20260923 14:49:31.387367 16613 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:49:31.388499 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 16613
I20260923 14:49:31.388597 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:49:31.388970 30731 external_mini_cluster.cc:1428] Setting key 1d124cbb413738c1a5e116810000de7b
Sep 23 14:49:31 dist-test-slave-wjf2 krb5kdc[16372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
I20260923 14:49:31.402827 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.389381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:49525 (local address 127.30.2.254:44799)
0923 14:49:31.389544 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:31.389548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:31.390724 (+ 1176us) server_negotiation.cc:407] Connection header received
0923 14:49:31.391572 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:31.391579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:31.391639 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:31.391729 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:31.393407 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.393917 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.395741 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.396051 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.396136 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.398864 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:31.398871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:31.398876 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:31.398903 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:31.400452 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:31.401081 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:31.401087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:31.401089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:31.401172 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:31.401513 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:31.401517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:31.401519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:31.401698 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:31.401824 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:31.402432 (+ 608us) server_negotiation.cc:299] Negotiation successful
0923 14:49:31.402571 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:49:31.403869 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.389763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:44799 (local address 127.30.2.194:49525)
0923 14:49:31.390558 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:31.390614 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:31.391370 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:31.391967 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:31.391973 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:31.392501 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:31.393243 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.393271 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.394049 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.394056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.395603 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.395618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.396215 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.396222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.396462 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.397085 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:49:31.397110 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:31.398681 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:31.400636 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:31.400641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:31.400652 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:31.400977 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:31.401280 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:31.401283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:31.401285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:31.401416 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:31.401824 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:31.401831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:31.402113 (+ 282us) client_negotiation.cc:769] Sending connection context
0923 14:49:31.402329 (+ 216us) client_negotiation.cc:240] Negotiation successful
0923 14:49:31.402684 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":123,"threads_started":1}
I20260923 14:49:31.405009 16742 heartbeater.cc:347] Connected to a master server at 127.30.2.254:44799
I20260923 14:49:31.405272 16742 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:31.405871 16742 heartbeater.cc:511] Master 127.30.2.254:44799 requested a full tablet report, sending...
I20260923 14:49:31.407114 16417 ts_manager.cc:194] Registered new tserver with Master: 3e284f77463349f98301e1ba4a775a5f (127.30.2.194:37651)
I20260923 14:49:31.408046 16417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:49525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:49:31.419092 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44799
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:31.530694 16747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:31.530937 16747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:31.530979 16747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:31.535099 16747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:31.535149 16747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:31.535246 16747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:49:31.540769 16747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:44799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:31.541790 16747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:31.542825 16747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:31.552495 16752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:31.552495 16753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:31.552528 16755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:31.552914 16747 server_base.cc:1034] running on GCE node
I20260923 14:49:31.553426 16747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:31.554087 16747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:31.555275 16747 hybrid_clock.cc:648] HybridClock initialized: now 1790174971555255 us; error 29 us; skew 500 ppm
Sep 23 14:49:31 dist-test-slave-wjf2 krb5kdc[16372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:31.558239 16747 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:49:31.559008 16747 webserver.cc:460] Webserver started at http://127.30.2.195:45859/ using document root <none> and password file <none>
I20260923 14:49:31.559496 16747 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:31.559551 16747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:31.559700 16747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:31.561606 16747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "8547ca713dd341c2b56b86ed44a2609a"
format_stamp: "Formatted at 2026-09-23 14:49:31 on dist-test-slave-wjf2"
server_key: "576aec0f564f5b43136e5fda7e7f161a"
server_key_iv: "c4cc5a86a1af4a2be3f5b90dfd63ce84"
server_key_version: "encryptionkey@0"
I20260923 14:49:31.562074 16747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "8547ca713dd341c2b56b86ed44a2609a"
format_stamp: "Formatted at 2026-09-23 14:49:31 on dist-test-slave-wjf2"
server_key: "576aec0f564f5b43136e5fda7e7f161a"
server_key_iv: "c4cc5a86a1af4a2be3f5b90dfd63ce84"
server_key_version: "encryptionkey@0"
I20260923 14:49:31.565529 16747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:49:31.567981 16762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:31.569190 16747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:31.569285 16747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "8547ca713dd341c2b56b86ed44a2609a"
format_stamp: "Formatted at 2026-09-23 14:49:31 on dist-test-slave-wjf2"
server_key: "576aec0f564f5b43136e5fda7e7f161a"
server_key_iv: "c4cc5a86a1af4a2be3f5b90dfd63ce84"
server_key_version: "encryptionkey@0"
I20260923 14:49:31.569406 16747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:31.581911 16747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:31.585031 16747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:31.585224 16747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:31.585758 16747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:31.586540 16747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:31.586586 16747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:31.586645 16747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:31.586678 16747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:31.595362 16747 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:42385
I20260923 14:49:31.595425 16875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:42385 every 8 connection(s)
I20260923 14:49:31.596231 16747 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:49:31.605121 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 16747
I20260923 14:49:31.605268 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:49:31.605638 30731 external_mini_cluster.cc:1428] Setting key 7d40c6257c657169394475f054553c30
Sep 23 14:49:31 dist-test-slave-wjf2 krb5kdc[16372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:49:31.610455 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.598185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:36917 (local address 127.30.2.254:44799)
0923 14:49:31.598351 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:31.598355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:31.599323 (+ 968us) server_negotiation.cc:407] Connection header received
0923 14:49:31.600379 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:31.600383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:31.600446 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:31.600552 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:31.601782 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.602338 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.603612 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.604017 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.604161 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.606959 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:31.606962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:31.606964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:31.606988 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:31.608550 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:31.608988 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:31.608991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:31.608992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:31.609061 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:31.609346 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:31.609348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:31.609349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:31.609517 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:31.609609 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:31.610242 (+ 633us) server_negotiation.cc:299] Negotiation successful
0923 14:49:31.610339 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 14:49:31.611335 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.598513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:44799 (local address 127.30.2.195:36917)
0923 14:49:31.599160 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:31.599198 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:31.600162 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:31.600727 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:31.600734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:31.601086 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:31.601626 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.601640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.602465 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.602484 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.603383 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.603392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.604198 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.604203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.604411 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.604973 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:49:31.604997 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:31.606765 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:31.608649 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:31.608654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:31.608664 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:31.608906 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:31.609139 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:31.609141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:31.609142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:31.609270 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:31.609674 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:31.609680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:31.609982 (+ 302us) client_negotiation.cc:769] Sending connection context
0923 14:49:31.610177 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 14:49:31.610464 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":108,"threads_started":1}
I20260923 14:49:31.612493 16876 heartbeater.cc:347] Connected to a master server at 127.30.2.254:44799
I20260923 14:49:31.612763 16876 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:31.613297 16876 heartbeater.cc:511] Master 127.30.2.254:44799 requested a full tablet report, sending...
I20260923 14:49:31.614406 16417 ts_manager.cc:194] Registered new tserver with Master: 8547ca713dd341c2b56b86ed44a2609a (127.30.2.195:42385)
I20260923 14:49:31.615043 16417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:36917
I20260923 14:49:31.620312 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:49:31.630023 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.622358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.30.2.254:44799)
0923 14:49:31.622470 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:31.622473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:31.622645 (+ 172us) server_negotiation.cc:407] Connection header received
0923 14:49:31.622740 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:31.622742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:31.622806 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:31.622874 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:31.623525 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.624113 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.624983 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.625188 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.625272 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.626804 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:31.626808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:31.626809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:31.626830 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:31.628272 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:31.628789 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:31.628793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:31.628794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:31.628881 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:31.629261 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:31.629264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:31.629275 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:31.629471 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:31.629587 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:31.629827 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 14:49:31.629904 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:49:31.632984 16417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:49:31.635947 16417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:49:31.651049 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.644854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.30.2.194:37651)
0923 14:49:31.645577 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:31.645589 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:31.645617 (+ 28us) server_negotiation.cc:407] Connection header received
0923 14:49:31.645702 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:31.645708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:31.645922 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:31.646547 (+ 625us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:31.646712 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.647860 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.649328 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.650301 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.650436 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.650600 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:31.650760 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 14:49:31.650873 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":128,"threads_started":1}
I20260923 14:49:31.651041 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.644709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:37651 (local address 127.0.0.1:55856)
0923 14:49:31.645057 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:31.645102 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:31.645227 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:31.646113 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:31.646122 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:31.646172 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:31.646350 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.646359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.647986 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.647992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.649186 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.649197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.650436 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.650441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.650554 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.650594 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 14:49:31.650721 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 14:49:31.650886 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":34,"spinlock_wait_cycles":9344,"thread_start_us":113,"threads_started":1}
I20260923 14:49:31.651070 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.645475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:42385 (local address 127.0.0.1:50200)
0923 14:49:31.646061 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:31.646074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:31.646169 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:31.647027 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:31.647030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:31.647048 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:31.647127 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.647132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.648580 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.648584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.649599 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.649609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.650552 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.650556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.650644 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.650670 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 14:49:31.650723 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 14:49:31.650882 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":5,"spinlock_wait_cycles":7552,"thread_start_us":95,"threads_started":1}
I20260923 14:49:31.651443 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.644937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:37869 (local address 127.0.0.1:49934)
0923 14:49:31.645585 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:31.645600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:31.645686 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:31.646787 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:31.646790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:31.646807 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:31.646928 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.646938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.648580 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.648584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.649956 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.649970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.651056 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.651059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.651168 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.651202 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 14:49:31.651255 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 14:49:31.651327 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":156,"thread_start_us":89,"threads_started":1}
I20260923 14:49:31.651997 16891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.645524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.30.2.193:37869)
0923 14:49:31.645888 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:31.646303 (+ 415us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:31.646330 (+ 27us) server_negotiation.cc:407] Connection header received
0923 14:49:31.646431 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:31.646437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:31.646634 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:31.646790 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:31.647065 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.648419 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.650106 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.650921 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.651514 (+ 593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.651685 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:31.651742 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 14:49:31.651853 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":87,"threads_started":1}
I20260923 14:49:31.652240 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.646005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.30.2.195:42385)
0923 14:49:31.646518 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:31.646527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:31.646545 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:49:31.646632 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:31.646637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:31.646829 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:31.647023 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:31.647254 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.648419 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.649730 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.650415 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.651825 (+ 1410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.651955 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:31.652006 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 14:49:31.652101 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":130,"threads_started":1}
I20260923 14:49:31.654644 16542 tablet_service.cc:1467] Processing CreateTablet for tablet bab0488be856421c835864568144b57e (DEFAULT_TABLE table=test-table [id=a338d78f136e4b75a8739a777274e7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:31.654709 16676 tablet_service.cc:1467] Processing CreateTablet for tablet bab0488be856421c835864568144b57e (DEFAULT_TABLE table=test-table [id=a338d78f136e4b75a8739a777274e7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:31.655289 16810 tablet_service.cc:1467] Processing CreateTablet for tablet bab0488be856421c835864568144b57e (DEFAULT_TABLE table=test-table [id=a338d78f136e4b75a8739a777274e7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:31.655730 16542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab0488be856421c835864568144b57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:31.655812 16676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab0488be856421c835864568144b57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:31.656286 16810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab0488be856421c835864568144b57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:31.662444 16893 tablet_bootstrap.cc:492] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588: Bootstrap starting.
I20260923 14:49:31.663128 16894 tablet_bootstrap.cc:492] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: Bootstrap starting.
I20260923 14:49:31.664213 16895 tablet_bootstrap.cc:492] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: Bootstrap starting.
I20260923 14:49:31.664827 16893 tablet_bootstrap.cc:654] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:31.665467 16894 tablet_bootstrap.cc:654] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:31.666219 16895 tablet_bootstrap.cc:654] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:31.666268 16893 log.cc:824] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:31.666733 16894 log.cc:824] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:31.667482 16895 log.cc:824] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:31.668504 16893 tablet_bootstrap.cc:492] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588: No bootstrap required, opened a new log
I20260923 14:49:31.668720 16894 tablet_bootstrap.cc:492] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: No bootstrap required, opened a new log
I20260923 14:49:31.668723 16893 ts_tablet_manager.cc:1397] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:49:31.668905 16894 ts_tablet_manager.cc:1397] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:49:31.669404 16895 tablet_bootstrap.cc:492] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: No bootstrap required, opened a new log
I20260923 14:49:31.669571 16895 ts_tablet_manager.cc:1397] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260923 14:49:31.671832 16893 raft_consensus.cc:348] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e284f77463349f98301e1ba4a775a5f" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37651 } } peers { permanent_uuid: "8547ca713dd341c2b56b86ed44a2609a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 42385 } } peers { permanent_uuid: "5081b5e5b2e2458198a0d64979004588" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37869 } }
I20260923 14:49:31.672086 16893 raft_consensus.cc:374] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:31.672140 16893 raft_consensus.cc:729] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5081b5e5b2e2458198a0d64979004588, State: Initialized, Role: FOLLOWER
I20260923 14:49:31.672637 16893 consensus_queue.cc:260] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e284f77463349f98301e1ba4a775a5f" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37651 } } peers { permanent_uuid: "8547ca713dd341c2b56b86ed44a2609a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 42385 } } peers { permanent_uuid: "5081b5e5b2e2458198a0d64979004588" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37869 } }
I20260923 14:49:31.672794 16894 raft_consensus.cc:348] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e284f77463349f98301e1ba4a775a5f" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37651 } } peers { permanent_uuid: "8547ca713dd341c2b56b86ed44a2609a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 42385 } } peers { permanent_uuid: "5081b5e5b2e2458198a0d64979004588" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37869 } }
I20260923 14:49:31.673127 16894 raft_consensus.cc:374] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:31.673194 16894 raft_consensus.cc:729] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e284f77463349f98301e1ba4a775a5f, State: Initialized, Role: FOLLOWER
I20260923 14:49:31.673378 16895 raft_consensus.cc:348] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e284f77463349f98301e1ba4a775a5f" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37651 } } peers { permanent_uuid: "8547ca713dd341c2b56b86ed44a2609a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 42385 } } peers { permanent_uuid: "5081b5e5b2e2458198a0d64979004588" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37869 } }
I20260923 14:49:31.673612 16895 raft_consensus.cc:374] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:31.673625 16608 heartbeater.cc:503] Master 127.30.2.254:44799 was elected leader, sending a full tablet report...
I20260923 14:49:31.673658 16895 raft_consensus.cc:729] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8547ca713dd341c2b56b86ed44a2609a, State: Initialized, Role: FOLLOWER
I20260923 14:49:31.673655 16894 consensus_queue.cc:260] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e284f77463349f98301e1ba4a775a5f" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37651 } } peers { permanent_uuid: "8547ca713dd341c2b56b86ed44a2609a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 42385 } } peers { permanent_uuid: "5081b5e5b2e2458198a0d64979004588" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37869 } }
I20260923 14:49:31.673870 16893 ts_tablet_manager.cc:1428] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:49:31.674484 16895 consensus_queue.cc:260] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e284f77463349f98301e1ba4a775a5f" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37651 } } peers { permanent_uuid: "8547ca713dd341c2b56b86ed44a2609a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 42385 } } peers { permanent_uuid: "5081b5e5b2e2458198a0d64979004588" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37869 } }
I20260923 14:49:31.674634 16894 ts_tablet_manager.cc:1428] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 14:49:31.674966 16742 heartbeater.cc:503] Master 127.30.2.254:44799 was elected leader, sending a full tablet report...
I20260923 14:49:31.675473 16895 ts_tablet_manager.cc:1428] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260923 14:49:31.675813 16876 heartbeater.cc:503] Master 127.30.2.254:44799 was elected leader, sending a full tablet report...
W20260923 14:49:31.847265 16877 tablet.cc:2369] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:49:31.888559 16743 tablet.cc:2369] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:49:31.903246 16901 raft_consensus.cc:482] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:49:31.903462 16901 raft_consensus.cc:504] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e284f77463349f98301e1ba4a775a5f" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37651 } } peers { permanent_uuid: "8547ca713dd341c2b56b86ed44a2609a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 42385 } } peers { permanent_uuid: "5081b5e5b2e2458198a0d64979004588" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37869 } }
I20260923 14:49:31.904685 16901 leader_election.cc:288] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e284f77463349f98301e1ba4a775a5f (127.30.2.194:37651), 5081b5e5b2e2458198a0d64979004588 (127.30.2.193:37869)
I20260923 14:49:31.909168 16891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.905085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:46491 (local address 127.30.2.193:37869)
0923 14:49:31.905205 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:31.905209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:31.905225 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:49:31.905396 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:31.905400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:31.905468 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:31.905555 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:31.905908 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.906662 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.908075 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.908581 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.908670 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.908712 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:31.908968 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 14:49:31.909025 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:49:31.909178 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.904985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:37869 (local address 127.30.2.195:46491)
0923 14:49:31.905118 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:31.905135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:31.905240 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:31.905614 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:31.905617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:31.905641 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:31.905753 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.905758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.906803 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.906807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.907941 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.907952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.908702 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.908706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.908823 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.908853 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 14:49:31.908936 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 14:49:31.909008 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":5}
I20260923 14:49:31.909582 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.905108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:60541 (local address 127.30.2.194:37651)
0923 14:49:31.905229 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:31.905233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:31.905897 (+ 664us) server_negotiation.cc:407] Connection header received
0923 14:49:31.906031 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:31.906034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:31.906082 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:31.906147 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:31.906422 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.907176 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.908418 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.908993 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.909112 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.909159 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:31.909376 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 14:49:31.909472 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:49:31.909934 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.904998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:37651 (local address 127.30.2.195:60541)
0923 14:49:31.905796 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:31.905807 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:31.905919 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:31.906177 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:31.906180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:31.906198 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:31.906312 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.906318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.907315 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.907317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.908288 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:31.908298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.909108 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.909112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:31.909215 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.909242 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 14:49:31.909723 (+ 481us) client_negotiation.cc:240] Negotiation successful
0923 14:49:31.909837 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":401,"thread_start_us":130,"threads_started":1}
I20260923 14:49:31.910072 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab0488be856421c835864568144b57e" candidate_uuid: "8547ca713dd341c2b56b86ed44a2609a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5081b5e5b2e2458198a0d64979004588" is_pre_election: true
I20260923 14:49:31.910444 16562 raft_consensus.cc:2454] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8547ca713dd341c2b56b86ed44a2609a in term 0.
I20260923 14:49:31.910682 16696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab0488be856421c835864568144b57e" candidate_uuid: "8547ca713dd341c2b56b86ed44a2609a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e284f77463349f98301e1ba4a775a5f" is_pre_election: true
I20260923 14:49:31.911000 16696 raft_consensus.cc:2454] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8547ca713dd341c2b56b86ed44a2609a in term 0.
I20260923 14:49:31.911055 16763 leader_election.cc:302] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5081b5e5b2e2458198a0d64979004588, 8547ca713dd341c2b56b86ed44a2609a; no voters:
I20260923 14:49:31.911409 16901 raft_consensus.cc:2781] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:49:31.911512 16901 raft_consensus.cc:482] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:49:31.911545 16901 raft_consensus.cc:3037] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:31.912653 16901 raft_consensus.cc:504] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e284f77463349f98301e1ba4a775a5f" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37651 } } peers { permanent_uuid: "8547ca713dd341c2b56b86ed44a2609a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 42385 } } peers { permanent_uuid: "5081b5e5b2e2458198a0d64979004588" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37869 } }
I20260923 14:49:31.913014 16901 leader_election.cc:288] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [CANDIDATE]: Term 1 election: Requested vote from peers 3e284f77463349f98301e1ba4a775a5f (127.30.2.194:37651), 5081b5e5b2e2458198a0d64979004588 (127.30.2.193:37869)
I20260923 14:49:31.913426 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab0488be856421c835864568144b57e" candidate_uuid: "8547ca713dd341c2b56b86ed44a2609a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5081b5e5b2e2458198a0d64979004588"
I20260923 14:49:31.913506 16696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab0488be856421c835864568144b57e" candidate_uuid: "8547ca713dd341c2b56b86ed44a2609a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e284f77463349f98301e1ba4a775a5f"
I20260923 14:49:31.913579 16562 raft_consensus.cc:3037] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:31.913645 16696 raft_consensus.cc:3037] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f [term 0 FOLLOWER]: Advancing to term 1
W20260923 14:49:31.913898 16609 tablet.cc:2369] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:49:31.914814 16562 raft_consensus.cc:2454] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8547ca713dd341c2b56b86ed44a2609a in term 1.
I20260923 14:49:31.914831 16696 raft_consensus.cc:2454] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8547ca713dd341c2b56b86ed44a2609a in term 1.
I20260923 14:49:31.915127 16763 leader_election.cc:302] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5081b5e5b2e2458198a0d64979004588, 8547ca713dd341c2b56b86ed44a2609a; no voters:
I20260923 14:49:31.915325 16901 raft_consensus.cc:2781] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:31.915546 16901 raft_consensus.cc:686] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 1 LEADER]: Becoming Leader. State: Replica: 8547ca713dd341c2b56b86ed44a2609a, State: Running, Role: LEADER
I20260923 14:49:31.915961 16901 consensus_queue.cc:237] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e284f77463349f98301e1ba4a775a5f" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37651 } } peers { permanent_uuid: "8547ca713dd341c2b56b86ed44a2609a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 42385 } } peers { permanent_uuid: "5081b5e5b2e2458198a0d64979004588" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37869 } }
I20260923 14:49:31.919297 16415 catalog_manager.cc:5495] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8547ca713dd341c2b56b86ed44a2609a (127.30.2.195). New cstate: current_term: 1 leader_uuid: "8547ca713dd341c2b56b86ed44a2609a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e284f77463349f98301e1ba4a775a5f" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8547ca713dd341c2b56b86ed44a2609a" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 42385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5081b5e5b2e2458198a0d64979004588" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37869 } health_report { overall_health: UNKNOWN } } }
I20260923 14:49:31.945331 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.941528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.30.2.195:42385)
0923 14:49:31.941668 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:31.941672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:31.941687 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:49:31.941891 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:31.941894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:31.941956 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:31.942072 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:49:31.942343 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.942885 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.943756 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:31.944028 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:31.944143 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:31.944461 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:49:31.944877 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:49:31.945040 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:31.945129 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 14:49:31.945167 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:49:31.951501 16696 raft_consensus.cc:1261] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f [term 1 FOLLOWER]: Refusing update from remote peer 8547ca713dd341c2b56b86ed44a2609a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:31.951501 16562 raft_consensus.cc:1261] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588 [term 1 FOLLOWER]: Refusing update from remote peer 8547ca713dd341c2b56b86ed44a2609a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:31.952342 16903 consensus_queue.cc:1035] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5081b5e5b2e2458198a0d64979004588" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 37869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:31.952554 16901 consensus_queue.cc:1035] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e284f77463349f98301e1ba4a775a5f" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:31.961073 16907 mvcc.cc:205] Tried to move back new op lower bound from 7332556685105942528 to 7332556684970614784. Current Snapshot: MvccSnapshot[applied={T|T < 7332556685105942528}]
I20260923 14:49:31.969084 16415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:31.969272 16415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:31.971737 16415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa9037d012194c37a05f11342cb3fa1e: Sending DeleteTablet for 3 replicas of tablet bab0488be856421c835864568144b57e
I20260923 14:49:31.972633 16676 tablet_service.cc:1514] Processing DeleteTablet for tablet bab0488be856421c835864568144b57e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:31 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:55856
I20260923 14:49:31.972633 16542 tablet_service.cc:1514] Processing DeleteTablet for tablet bab0488be856421c835864568144b57e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:31 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:49934
I20260923 14:49:31.972746 16810 tablet_service.cc:1514] Processing DeleteTablet for tablet bab0488be856421c835864568144b57e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:31 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:50200
I20260923 14:49:31.973152 16916 tablet_replica.cc:330] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588: stopping tablet replica
I20260923 14:49:31.973330 16917 tablet_replica.cc:330] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: stopping tablet replica
I20260923 14:49:31.973595 16916 raft_consensus.cc:2229] T bab0488be856421c835864568144b57e P 5081b5e5b2e2458198a0d64979004588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:49:31.973640 16917 raft_consensus.cc:2229] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:49:31.973977 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 16479
I20260923 14:49:31.974185 16917 raft_consensus.cc:2258] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:31.976322 16917 ts_tablet_manager.cc:1905] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:49:31.977761 16915 tablet_replica.cc:330] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: stopping tablet replica
I20260923 14:49:31.978114 16915 raft_consensus.cc:2229] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:49:31.978462 16915 raft_consensus.cc:2258] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:31.979607 16917 ts_tablet_manager.cc:1918] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:49:31.979750 16917 log.cc:1196] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/bab0488be856421c835864568144b57e
I20260923 14:49:31.980070 16915 ts_tablet_manager.cc:1905] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:49:31.980262 16917 ts_tablet_manager.cc:1939] T bab0488be856421c835864568144b57e P 8547ca713dd341c2b56b86ed44a2609a: Deleting consensus metadata
I20260923 14:49:31.981556 16403 catalog_manager.cc:4841] TS 8547ca713dd341c2b56b86ed44a2609a (127.30.2.195:42385): tablet bab0488be856421c835864568144b57e (table test-table [id=a338d78f136e4b75a8739a777274e7d6]) successfully deleted
I20260923 14:49:31.982590 16915 ts_tablet_manager.cc:1918] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:49:31.982702 16915 log.cc:1196] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/bab0488be856421c835864568144b57e
I20260923 14:49:31.983052 16915 ts_tablet_manager.cc:1939] T bab0488be856421c835864568144b57e P 3e284f77463349f98301e1ba4a775a5f: Deleting consensus metadata
W20260923 14:49:31.983256 16401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:37869 (error 108)
I20260923 14:49:31.983449 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 16613
I20260923 14:49:31.984685 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.984366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:37869 (local address 127.0.0.1:49944)
0923 14:49:31.984542 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:31.984604 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260923 14:49:31.984931 16401 catalog_manager.cc:4584] TS 5081b5e5b2e2458198a0d64979004588 (127.30.2.193:37869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bab0488be856421c835864568144b57e: Network error: Client connection negotiation failed: client connection to 127.30.2.193:37869: connect: Connection refused (error 111)
I20260923 14:49:31.989795 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 16747
I20260923 14:49:31.990016 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:31.989736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:37651 (local address 127.0.0.1:55870)
0923 14:49:31.989903 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:31.989947 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.194:37651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 14:49:31.995734 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 16385
2026-09-23T14:49:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2472 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T14:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:32Z Disabled control of system clock
I20260923 14:49:32.027148 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:39017
--webserver_interface=127.30.2.254
--webserver_port=0
--builtin_ntp_servers=127.30.2.212:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:39017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:49:32.138458 16922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:32.138731 16922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:32.138777 16922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:32.143093 16922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:32.143143 16922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:32.143158 16922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:32.143175 16922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:32.143190 16922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:32.147891 16922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:39017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:39017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:32.148872 16922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:32.150224 16922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:32.158876 16927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:32.158946 16922 server_base.cc:1034] running on GCE node
W20260923 14:49:32.158907 16928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:32.158893 16930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:32.159690 16922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:32.160916 16922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:32.164196 16922 hybrid_clock.cc:648] HybridClock initialized: now 1790174972164165 us; error 40 us; skew 500 ppm
I20260923 14:49:32.166002 16922 webserver.cc:460] Webserver started at http://127.30.2.254:43253/ using document root <none> and password file <none>
I20260923 14:49:32.166610 16922 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:32.166704 16922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:32.166942 16922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:32.169023 16922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "ba1823b58b1d4a8294a24902476459e4"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "1f26e252f80f4b8595f7f88074ae6be1"
server_key_iv: "7ff4b1d75269787af52005f013d75717"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.169624 16922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "ba1823b58b1d4a8294a24902476459e4"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "1f26e252f80f4b8595f7f88074ae6be1"
server_key_iv: "7ff4b1d75269787af52005f013d75717"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.173228 16922 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:49:32.175555 16936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:32.176712 16922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:32.176887 16922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "ba1823b58b1d4a8294a24902476459e4"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "1f26e252f80f4b8595f7f88074ae6be1"
server_key_iv: "7ff4b1d75269787af52005f013d75717"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.176993 16922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:32.198697 16922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:32.199355 16922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:32.199545 16922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:32.206204 16922 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:39017
I20260923 14:49:32.206226 16988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:39017 every 8 connection(s)
I20260923 14:49:32.207113 16922 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:32.210152 16989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:32.212800 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 16922
I20260923 14:49:32.212971 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:32.213301 30731 external_mini_cluster.cc:1428] Setting key 350cc878d22561afbfddd2aa5e8441cb
I20260923 14:49:32.216064 16989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4: Bootstrap starting.
I20260923 14:49:32.218176 16989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:32.218916 16989 log.cc:824] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:32.220777 16989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4: No bootstrap required, opened a new log
I20260923 14:49:32.223258 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.214726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.30.2.254:39017)
0923 14:49:32.215366 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:32.215383 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:32.215418 (+ 35us) server_negotiation.cc:407] Connection header received
0923 14:49:32.216097 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:32.216137 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:32.216546 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:32.216927 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:32.217468 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.218517 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.219548 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.220115 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.220328 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.221270 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:32.221276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:32.221289 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:32.221669 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:32.221725 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:32.221798 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:32.221911 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:32.222107 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 14:49:32.222400 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":117,"threads_started":1}
I20260923 14:49:32.223878 16989 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1823b58b1d4a8294a24902476459e4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39017 } }
I20260923 14:49:32.224100 16989 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:32.224170 16989 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1823b58b1d4a8294a24902476459e4, State: Initialized, Role: FOLLOWER
I20260923 14:49:32.224638 16989 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1823b58b1d4a8294a24902476459e4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39017 } }
I20260923 14:49:32.224784 16989 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:32.224864 16989 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:32.224982 16989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:32.225756 16989 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1823b58b1d4a8294a24902476459e4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39017 } }
I20260923 14:49:32.226092 16989 leader_election.cc:302] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba1823b58b1d4a8294a24902476459e4; no voters:
I20260923 14:49:32.226378 16989 leader_election.cc:288] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:32.226552 16994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:32.226794 16994 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [term 1 LEADER]: Becoming Leader. State: Replica: ba1823b58b1d4a8294a24902476459e4, State: Running, Role: LEADER
I20260923 14:49:32.227092 16994 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1823b58b1d4a8294a24902476459e4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39017 } }
I20260923 14:49:32.227569 16989 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:32.228720 16995 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba1823b58b1d4a8294a24902476459e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1823b58b1d4a8294a24902476459e4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39017 } } }
I20260923 14:49:32.228854 16995 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:32.229194 16996 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba1823b58b1d4a8294a24902476459e4. Latest consensus state: current_term: 1 leader_uuid: "ba1823b58b1d4a8294a24902476459e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1823b58b1d4a8294a24902476459e4" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 39017 } } }
I20260923 14:49:32.229350 16996 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4 [sys.catalog]: This master's current role is: LEADER
W20260923 14:49:32.232707 17009 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:49:32.232785 17009 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:49:32.232892 17003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:32.233858 17003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:32.239465 17003 catalog_manager.cc:1347] Generated new cluster ID: a4b44daa58c94a8a9c217885d85dc017
I20260923 14:49:32.239529 17003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:32.247977 17003 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:32.248819 17003 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:32.256247 17003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4: Generated new TSK 0
I20260923 14:49:32.256944 17003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:32.265182 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39017
--builtin_ntp_servers=127.30.2.212:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:49:32.375506 17013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:32.375759 17013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:32.375859 17013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:32.380434 17013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:32.380501 17013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:32.380585 17013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:49:32.385213 17013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:32.386194 17013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:32.387517 17013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:32.396759 17021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:32.396837 17018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:32.396759 17019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:32.397081 17013 server_base.cc:1034] running on GCE node
I20260923 14:49:32.397455 17013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:32.398108 17013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:32.399292 17013 hybrid_clock.cc:648] HybridClock initialized: now 1790174972399277 us; error 25 us; skew 500 ppm
I20260923 14:49:32.401098 17013 webserver.cc:460] Webserver started at http://127.30.2.193:37489/ using document root <none> and password file <none>
I20260923 14:49:32.401654 17013 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:32.401747 17013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:32.401965 17013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:32.403980 17013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "3dc282a2582a46c296fe5e3702896b0d"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "81ca4909a75a7cfe424cb00db7b469a4"
server_key_iv: "a58b23f1c09a2182431c00b363413377"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.404521 17013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "3dc282a2582a46c296fe5e3702896b0d"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "81ca4909a75a7cfe424cb00db7b469a4"
server_key_iv: "a58b23f1c09a2182431c00b363413377"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.408125 17013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:49:32.410487 17027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:32.411918 17013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:49:32.412050 17013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "3dc282a2582a46c296fe5e3702896b0d"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "81ca4909a75a7cfe424cb00db7b469a4"
server_key_iv: "a58b23f1c09a2182431c00b363413377"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.412156 17013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:32.423260 17013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:32.423926 17013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:32.424111 17013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:32.424667 17013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:32.425597 17013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:32.425668 17013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:32.425731 17013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:32.425774 17013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:32.434603 17013 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:40807
I20260923 14:49:32.434666 17140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:40807 every 8 connection(s)
I20260923 14:49:32.435551 17013 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:49:32.441720 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 17013
I20260923 14:49:32.441949 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:49:32.442317 30731 external_mini_cluster.cc:1428] Setting key abe063238d7056d468669a279d9e438e
I20260923 14:49:32.444608 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39017
--builtin_ntp_servers=127.30.2.212:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 14:49:32.445837 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.437545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:40077 (local address 127.30.2.254:39017)
0923 14:49:32.437706 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:32.437711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:32.438936 (+ 1225us) server_negotiation.cc:407] Connection header received
0923 14:49:32.440361 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:32.440364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:32.440413 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:32.440493 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:32.441372 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.442005 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.443517 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.443854 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.443960 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.445343 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:32.445346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:32.445348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:32.445360 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:32.445378 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:32.445386 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:32.445445 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:32.445666 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 14:49:32.445720 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:49:32.447064 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.438054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:39017 (local address 127.30.2.193:40077)
0923 14:49:32.438765 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:32.438805 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:32.440133 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:32.440661 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:32.440668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:32.440752 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:32.441245 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.441257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.442150 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.442154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.443390 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.443400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.444026 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.444031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.444249 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.444999 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:49:32.445003 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:32.445060 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:49:32.445148 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:32.445479 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:32.445491 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:32.445500 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 14:49:32.445633 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 14:49:32.445948 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":126,"threads_started":1}
I20260923 14:49:32.448572 17141 heartbeater.cc:347] Connected to a master server at 127.30.2.254:39017
I20260923 14:49:32.448843 17141 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:32.449584 17141 heartbeater.cc:511] Master 127.30.2.254:39017 requested a full tablet report, sending...
I20260923 14:49:32.451503 16953 ts_manager.cc:194] Registered new tserver with Master: 3dc282a2582a46c296fe5e3702896b0d (127.30.2.193:40807)
I20260923 14:49:32.452812 16953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.193:40077
W20260923 14:49:32.559115 17144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:32.559412 17144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:32.559480 17144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:32.563565 17144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:32.563648 17144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:32.563740 17144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:49:32.568499 17144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:32.569598 17144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:32.571089 17144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:32.580446 17150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:32.580457 17152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:32.580570 17149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:32.580654 17144 server_base.cc:1034] running on GCE node
I20260923 14:49:32.581130 17144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:32.581730 17144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:32.582928 17144 hybrid_clock.cc:648] HybridClock initialized: now 1790174972582911 us; error 33 us; skew 500 ppm
I20260923 14:49:32.584692 17144 webserver.cc:460] Webserver started at http://127.30.2.194:32975/ using document root <none> and password file <none>
I20260923 14:49:32.585289 17144 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:32.585376 17144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:32.585606 17144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:32.587566 17144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "170d7c1a8cc24e2e9003ddc91a2706bf"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "1da2213add94885950199059f4f52c00"
server_key_iv: "4c83825ad474c02b31228b4da18656dc"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.588171 17144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "170d7c1a8cc24e2e9003ddc91a2706bf"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "1da2213add94885950199059f4f52c00"
server_key_iv: "4c83825ad474c02b31228b4da18656dc"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.591631 17144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:49:32.594122 17158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:32.595280 17144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:49:32.595405 17144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "170d7c1a8cc24e2e9003ddc91a2706bf"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "1da2213add94885950199059f4f52c00"
server_key_iv: "4c83825ad474c02b31228b4da18656dc"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.595505 17144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:32.617345 17144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:32.618059 17144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:32.618268 17144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:32.618911 17144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:32.619968 17144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:32.620033 17144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:32.620095 17144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:32.620143 17144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:32.629237 17144 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:39523
I20260923 14:49:32.629302 17273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:39523 every 8 connection(s)
I20260923 14:49:32.630237 17144 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:49:32.631911 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 17144
I20260923 14:49:32.632046 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:49:32.632401 30731 external_mini_cluster.cc:1428] Setting key 37880b10f7bea2737a33ba73dedf062a
I20260923 14:49:32.634509 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39017
--builtin_ntp_servers=127.30.2.212:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 14:49:32.641378 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.633777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:37071 (local address 127.30.2.254:39017)
0923 14:49:32.633904 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:32.633909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:32.633923 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:49:32.635257 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:32.635261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:32.635312 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:32.635417 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:32.636660 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.637205 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.639118 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.639383 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.639499 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.640903 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:32.640906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:32.640908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:32.640920 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:32.640940 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:32.640949 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:32.641013 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:32.641211 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 14:49:32.641265 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 14:49:32.642685 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.632770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:39017 (local address 127.30.2.194:37071)
0923 14:49:32.633607 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:32.633661 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:32.635047 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:32.635623 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:32.635634 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:32.635762 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:32.636497 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.636518 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.637329 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.637336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.638908 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.638924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.639569 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.639577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.639879 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.640560 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:49:32.640564 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:32.640627 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:49:32.640720 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:32.641039 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:32.641052 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:32.641060 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 14:49:32.641201 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 14:49:32.641531 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":146,"threads_started":1}
I20260923 14:49:32.643997 17274 heartbeater.cc:347] Connected to a master server at 127.30.2.254:39017
I20260923 14:49:32.644243 17274 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:32.644764 17274 heartbeater.cc:511] Master 127.30.2.254:39017 requested a full tablet report, sending...
I20260923 14:49:32.645833 16953 ts_manager.cc:194] Registered new tserver with Master: 170d7c1a8cc24e2e9003ddc91a2706bf (127.30.2.194:39523)
I20260923 14:49:32.646415 16953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.194:37071
W20260923 14:49:32.747421 17277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:32.747660 17277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:32.747725 17277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:32.751659 17277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:32.751708 17277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:32.751833 17277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:49:32.756493 17277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:39017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:32.757536 17277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:32.758915 17277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:32.768759 17283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:32.768786 17285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:32.768759 17282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:32.769017 17277 server_base.cc:1034] running on GCE node
I20260923 14:49:32.769541 17277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:32.770126 17277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:32.771332 17277 hybrid_clock.cc:648] HybridClock initialized: now 1790174972771297 us; error 51 us; skew 500 ppm
I20260923 14:49:32.773175 17277 webserver.cc:460] Webserver started at http://127.30.2.195:42387/ using document root <none> and password file <none>
I20260923 14:49:32.773777 17277 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:32.773872 17277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:32.774151 17277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:32.776266 17277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "4b8a14b89acb46db87a6c9a435992405"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "ec752b39228f23603c5aaf2217f3826c"
server_key_iv: "b578f5adca85ce76347908167e7cf49e"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.776932 17277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "4b8a14b89acb46db87a6c9a435992405"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "ec752b39228f23603c5aaf2217f3826c"
server_key_iv: "b578f5adca85ce76347908167e7cf49e"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.780912 17277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 14:49:32.783389 17291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:32.784600 17277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:32.784737 17277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "4b8a14b89acb46db87a6c9a435992405"
format_stamp: "Formatted at 2026-09-23 14:49:32 on dist-test-slave-wjf2"
server_key: "ec752b39228f23603c5aaf2217f3826c"
server_key_iv: "b578f5adca85ce76347908167e7cf49e"
server_key_version: "encryptionkey@0"
I20260923 14:49:32.784873 17277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:32.796095 17277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:32.796732 17277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:32.796931 17277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:32.797500 17277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:32.798477 17277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:32.798547 17277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:32.798614 17277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:32.798648 17277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:32.807447 17277 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:33275
I20260923 14:49:32.807551 17404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:33275 every 8 connection(s)
I20260923 14:49:32.808382 17277 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:49:32.811151 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 17277
I20260923 14:49:32.811282 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:49:32.811583 30731 external_mini_cluster.cc:1428] Setting key c65f011308a5094a167085083dd9a846
I20260923 14:49:32.818625 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.810325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:41065 (local address 127.30.2.254:39017)
0923 14:49:32.810500 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:32.810504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:32.811669 (+ 1165us) server_negotiation.cc:407] Connection header received
0923 14:49:32.813340 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:32.813343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:32.813403 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:32.813519 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:32.814471 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.815068 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.816379 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.816713 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.816837 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.818099 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:32.818103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:32.818106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:32.818119 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:32.818147 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:32.818156 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:32.818238 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:32.818414 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 14:49:32.818476 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 14:49:32.819547 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.810692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:39017 (local address 127.30.2.195:41065)
0923 14:49:32.811509 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:32.811553 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:32.813108 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:32.813706 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:32.813714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:32.813812 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:32.814316 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.814329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.815200 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.815204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.816243 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.816254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.816923 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.816929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.817127 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.817815 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:49:32.817817 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:32.817858 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:49:32.817922 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:32.818241 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:32.818254 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:32.818262 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 14:49:32.818414 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 14:49:32.818704 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":147,"threads_started":1}
I20260923 14:49:32.820739 17405 heartbeater.cc:347] Connected to a master server at 127.30.2.254:39017
I20260923 14:49:32.820988 17405 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:32.821449 17405 heartbeater.cc:511] Master 127.30.2.254:39017 requested a full tablet report, sending...
I20260923 14:49:32.822425 16953 ts_manager.cc:194] Registered new tserver with Master: 4b8a14b89acb46db87a6c9a435992405 (127.30.2.195:33275)
I20260923 14:49:32.823166 16953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.195:41065
I20260923 14:49:32.825294 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:49:32.831418 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.827078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.30.2.254:39017)
0923 14:49:32.827213 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:32.827218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:32.827400 (+ 182us) server_negotiation.cc:407] Connection header received
0923 14:49:32.827483 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:32.827486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:32.827529 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:32.827616 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:32.827971 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.828772 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.829708 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.829947 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.830058 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.830934 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:32.830938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:32.830940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:32.830960 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:32.830976 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:32.830984 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:32.831111 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:32.831248 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 14:49:32.831282 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:49:32.833988 16953 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:49:32.836895 16953 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:49:32.851225 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.845837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.30.2.195:33275)
0923 14:49:32.846822 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:32.846845 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:32.846867 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:49:32.846934 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:32.846938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:32.847113 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:32.847512 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:32.847638 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.848720 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.849965 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.850543 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.850668 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.850860 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:32.850979 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 14:49:32.851073 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":135,"threads_started":1}
I20260923 14:49:32.851219 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.845166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33275 (local address 127.0.0.1:38706)
0923 14:49:32.845580 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:32.845623 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:32.845736 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:32.847254 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:32.847257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:32.847276 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:32.847366 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.847372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.848824 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.848828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.849802 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.849811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.850669 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.850673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.850782 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.850820 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 14:49:32.850945 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 14:49:32.851062 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":23,"thread_start_us":96,"threads_started":1}
I20260923 14:49:32.851616 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.845976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:40807 (local address 127.0.0.1:46484)
0923 14:49:32.846237 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:32.846248 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:32.846318 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:32.847044 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:32.847047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:32.847063 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:32.847243 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.847254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.848377 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.848383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.850023 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.850031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.851035 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.851038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.851122 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.851140 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 14:49:32.851181 (+ 41us) client_negotiation.cc:240] Negotiation successful
0923 14:49:32.851237 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":62,"thread_start_us":119,"threads_started":1}
I20260923 14:49:32.851634 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.845273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:39523 (local address 127.0.0.1:51372)
0923 14:49:32.845958 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:32.845975 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:32.846065 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:32.846837 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:32.846845 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:32.846870 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:32.847002 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.847008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.848634 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.848637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.849845 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.849854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.850911 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.850915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.851020 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.851046 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 14:49:32.851093 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 14:49:32.851161 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":71,"thread_start_us":100,"threads_started":1}
I20260923 14:49:32.851843 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.845463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.30.2.194:39523)
0923 14:49:32.846282 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:32.846294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:32.846319 (+ 25us) server_negotiation.cc:407] Connection header received
0923 14:49:32.846418 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:32.846424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:32.846653 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:32.847154 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:32.847307 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.848512 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.849977 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.850757 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.851292 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.851478 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:32.851533 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 14:49:32.851627 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":116,"threads_started":1}
I20260923 14:49:32.852381 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.846259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.30.2.193:40807)
0923 14:49:32.846541 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:32.846549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:32.846569 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:49:32.846653 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:32.846658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:32.846890 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:32.847388 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:32.847498 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.848264 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.850142 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.850895 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.851900 (+ 1005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.852082 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:32.852132 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 14:49:32.852219 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
I20260923 14:49:32.854421 17339 tablet_service.cc:1467] Processing CreateTablet for tablet 6278caf5e3174822a1c1b767d651c8a4 (DEFAULT_TABLE table=test-table [id=c903cf748c1f4d7ab4dfc743ea571af4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:32.855414 17339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6278caf5e3174822a1c1b767d651c8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:32.855031 17207 tablet_service.cc:1467] Processing CreateTablet for tablet 6278caf5e3174822a1c1b767d651c8a4 (DEFAULT_TABLE table=test-table [id=c903cf748c1f4d7ab4dfc743ea571af4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:32.855024 17075 tablet_service.cc:1467] Processing CreateTablet for tablet 6278caf5e3174822a1c1b767d651c8a4 (DEFAULT_TABLE table=test-table [id=c903cf748c1f4d7ab4dfc743ea571af4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:32.856043 17207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6278caf5e3174822a1c1b767d651c8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:32.856045 17075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6278caf5e3174822a1c1b767d651c8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:32.861745 17422 tablet_bootstrap.cc:492] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405: Bootstrap starting.
I20260923 14:49:32.863686 17423 tablet_bootstrap.cc:492] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d: Bootstrap starting.
I20260923 14:49:32.864032 17422 tablet_bootstrap.cc:654] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:32.864082 17424 tablet_bootstrap.cc:492] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: Bootstrap starting.
I20260923 14:49:32.865110 17422 log.cc:824] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:32.865911 17423 tablet_bootstrap.cc:654] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:32.865911 17424 tablet_bootstrap.cc:654] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:32.866729 17423 log.cc:824] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:32.866729 17424 log.cc:824] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:32.866925 17422 tablet_bootstrap.cc:492] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405: No bootstrap required, opened a new log
I20260923 14:49:32.867143 17422 ts_tablet_manager.cc:1397] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:49:32.869139 17424 tablet_bootstrap.cc:492] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: No bootstrap required, opened a new log
I20260923 14:49:32.869205 17423 tablet_bootstrap.cc:492] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d: No bootstrap required, opened a new log
I20260923 14:49:32.869331 17424 ts_tablet_manager.cc:1397] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:49:32.869334 17423 ts_tablet_manager.cc:1397] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260923 14:49:32.870882 17422 raft_consensus.cc:348] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8a14b89acb46db87a6c9a435992405" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33275 } } peers { permanent_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 39523 } } peers { permanent_uuid: "3dc282a2582a46c296fe5e3702896b0d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40807 } }
I20260923 14:49:32.871111 17422 raft_consensus.cc:374] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:32.871161 17422 raft_consensus.cc:729] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8a14b89acb46db87a6c9a435992405, State: Initialized, Role: FOLLOWER
I20260923 14:49:32.871598 17422 consensus_queue.cc:260] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8a14b89acb46db87a6c9a435992405" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33275 } } peers { permanent_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 39523 } } peers { permanent_uuid: "3dc282a2582a46c296fe5e3702896b0d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40807 } }
I20260923 14:49:32.872227 17405 heartbeater.cc:503] Master 127.30.2.254:39017 was elected leader, sending a full tablet report...
I20260923 14:49:32.873188 17423 raft_consensus.cc:348] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8a14b89acb46db87a6c9a435992405" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33275 } } peers { permanent_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 39523 } } peers { permanent_uuid: "3dc282a2582a46c296fe5e3702896b0d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40807 } }
I20260923 14:49:32.873232 17424 raft_consensus.cc:348] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8a14b89acb46db87a6c9a435992405" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33275 } } peers { permanent_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 39523 } } peers { permanent_uuid: "3dc282a2582a46c296fe5e3702896b0d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40807 } }
I20260923 14:49:32.873431 17424 raft_consensus.cc:374] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:32.873431 17423 raft_consensus.cc:374] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:32.873481 17423 raft_consensus.cc:729] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc282a2582a46c296fe5e3702896b0d, State: Initialized, Role: FOLLOWER
I20260923 14:49:32.873489 17424 raft_consensus.cc:729] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170d7c1a8cc24e2e9003ddc91a2706bf, State: Initialized, Role: FOLLOWER
I20260923 14:49:32.873531 17422 ts_tablet_manager.cc:1428] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260923 14:49:32.873937 17424 consensus_queue.cc:260] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8a14b89acb46db87a6c9a435992405" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33275 } } peers { permanent_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 39523 } } peers { permanent_uuid: "3dc282a2582a46c296fe5e3702896b0d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40807 } }
I20260923 14:49:32.873937 17423 consensus_queue.cc:260] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8a14b89acb46db87a6c9a435992405" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33275 } } peers { permanent_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 39523 } } peers { permanent_uuid: "3dc282a2582a46c296fe5e3702896b0d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40807 } }
I20260923 14:49:32.874567 17274 heartbeater.cc:503] Master 127.30.2.254:39017 was elected leader, sending a full tablet report...
I20260923 14:49:32.874876 17141 heartbeater.cc:503] Master 127.30.2.254:39017 was elected leader, sending a full tablet report...
I20260923 14:49:32.875694 17423 ts_tablet_manager.cc:1428] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 14:49:32.876236 17424 ts_tablet_manager.cc:1428] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260923 14:49:32.881111 17275 tablet.cc:2369] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:49:32.936828 17142 tablet.cc:2369] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:49:32.969519 17430 raft_consensus.cc:482] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:49:32.969732 17430 raft_consensus.cc:504] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8a14b89acb46db87a6c9a435992405" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33275 } } peers { permanent_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 39523 } } peers { permanent_uuid: "3dc282a2582a46c296fe5e3702896b0d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40807 } }
I20260923 14:49:32.970973 17430 leader_election.cc:288] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b8a14b89acb46db87a6c9a435992405 (127.30.2.195:33275), 170d7c1a8cc24e2e9003ddc91a2706bf (127.30.2.194:39523)
I20260923 14:49:32.974748 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.971297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:33957 (local address 127.30.2.195:33275)
0923 14:49:32.971423 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:32.971428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:32.971489 (+ 61us) server_negotiation.cc:407] Connection header received
0923 14:49:32.971796 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:32.971802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:32.971859 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:32.971936 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:32.972205 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.972730 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.973815 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.974207 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.974338 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.974390 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:32.974561 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 14:49:32.974630 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:49:32.974781 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.971123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:33275 (local address 127.30.2.193:33957)
0923 14:49:32.971405 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:32.971420 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:32.971650 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:32.971963 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:32.971966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:32.971981 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:32.972099 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.972103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.972855 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.972858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.973677 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.973686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.974338 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.974342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.974442 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.974460 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 14:49:32.974546 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 14:49:32.974615 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":116,"threads_started":1}
I20260923 14:49:32.974812 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.971453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:43817 (local address 127.30.2.194:39523)
0923 14:49:32.971577 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:32.971580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:32.971685 (+ 105us) server_negotiation.cc:407] Connection header received
0923 14:49:32.971889 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:32.971892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:32.971937 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:32.972019 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:32.972293 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.972771 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.973872 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.974272 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.974400 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.974445 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:32.974630 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 14:49:32.974692 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:49:32.975239 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:32.971262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:39523 (local address 127.30.2.193:43817)
0923 14:49:32.971597 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:32.971608 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:32.971726 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:32.972033 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:32.972036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:32.972053 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:32.972172 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.972176 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.972883 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.972888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.973745 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:32.973754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:32.974395 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:32.974399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:32.974509 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:32.974526 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:49:32.974613 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 14:49:32.974680 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260923 14:49:32.975399 17359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6278caf5e3174822a1c1b767d651c8a4" candidate_uuid: "3dc282a2582a46c296fe5e3702896b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8a14b89acb46db87a6c9a435992405" is_pre_election: true
I20260923 14:49:32.975726 17359 raft_consensus.cc:2454] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc282a2582a46c296fe5e3702896b0d in term 0.
I20260923 14:49:32.975935 17227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6278caf5e3174822a1c1b767d651c8a4" candidate_uuid: "3dc282a2582a46c296fe5e3702896b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" is_pre_election: true
I20260923 14:49:32.976219 17227 raft_consensus.cc:2454] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc282a2582a46c296fe5e3702896b0d in term 0.
I20260923 14:49:32.976213 17031 leader_election.cc:302] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc282a2582a46c296fe5e3702896b0d, 4b8a14b89acb46db87a6c9a435992405; no voters:
I20260923 14:49:32.976508 17430 raft_consensus.cc:2781] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:49:32.976605 17430 raft_consensus.cc:482] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:49:32.976671 17430 raft_consensus.cc:3037] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:32.977811 17430 raft_consensus.cc:504] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8a14b89acb46db87a6c9a435992405" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33275 } } peers { permanent_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 39523 } } peers { permanent_uuid: "3dc282a2582a46c296fe5e3702896b0d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40807 } }
I20260923 14:49:32.978175 17430 leader_election.cc:288] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [CANDIDATE]: Term 1 election: Requested vote from peers 4b8a14b89acb46db87a6c9a435992405 (127.30.2.195:33275), 170d7c1a8cc24e2e9003ddc91a2706bf (127.30.2.194:39523)
I20260923 14:49:32.978463 17359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6278caf5e3174822a1c1b767d651c8a4" candidate_uuid: "3dc282a2582a46c296fe5e3702896b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8a14b89acb46db87a6c9a435992405"
I20260923 14:49:32.978564 17227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6278caf5e3174822a1c1b767d651c8a4" candidate_uuid: "3dc282a2582a46c296fe5e3702896b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf"
I20260923 14:49:32.978575 17359 raft_consensus.cc:3037] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:32.978688 17227 raft_consensus.cc:3037] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:32.979591 17359 raft_consensus.cc:2454] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc282a2582a46c296fe5e3702896b0d in term 1.
I20260923 14:49:32.979903 17031 leader_election.cc:302] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc282a2582a46c296fe5e3702896b0d, 4b8a14b89acb46db87a6c9a435992405; no voters:
I20260923 14:49:32.979929 17227 raft_consensus.cc:2454] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc282a2582a46c296fe5e3702896b0d in term 1.
I20260923 14:49:32.980057 17430 raft_consensus.cc:2781] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:32.980284 17430 raft_consensus.cc:686] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [term 1 LEADER]: Becoming Leader. State: Replica: 3dc282a2582a46c296fe5e3702896b0d, State: Running, Role: LEADER
I20260923 14:49:32.980633 17430 consensus_queue.cc:237] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8a14b89acb46db87a6c9a435992405" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33275 } } peers { permanent_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 39523 } } peers { permanent_uuid: "3dc282a2582a46c296fe5e3702896b0d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40807 } }
I20260923 14:49:32.983749 16951 catalog_manager.cc:5495] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dc282a2582a46c296fe5e3702896b0d (127.30.2.193). New cstate: current_term: 1 leader_uuid: "3dc282a2582a46c296fe5e3702896b0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8a14b89acb46db87a6c9a435992405" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 39523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc282a2582a46c296fe5e3702896b0d" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 40807 } health_report { overall_health: HEALTHY } } }
I20260923 14:49:33.007084 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:33.003284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.30.2.193:40807)
0923 14:49:33.003427 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:33.003431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:33.003445 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:49:33.003513 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:33.003515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:33.003572 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:33.003664 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:49:33.004049 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:33.004551 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:33.005404 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:33.005615 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:33.005694 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:33.006151 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:49:33.006610 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:49:33.006735 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:33.006930 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 14:49:33.006967 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:49:33.013829 17227 raft_consensus.cc:1261] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf [term 1 FOLLOWER]: Refusing update from remote peer 3dc282a2582a46c296fe5e3702896b0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:33.013829 17359 raft_consensus.cc:1261] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405 [term 1 FOLLOWER]: Refusing update from remote peer 3dc282a2582a46c296fe5e3702896b0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:33.014662 17433 consensus_queue.cc:1035] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b8a14b89acb46db87a6c9a435992405" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 33275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:33.014886 17430 consensus_queue.cc:1035] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "170d7c1a8cc24e2e9003ddc91a2706bf" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 39523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:33.022790 17436 mvcc.cc:205] Tried to move back new op lower bound from 7332556689455874048 to 7332556689331187712. Current Snapshot: MvccSnapshot[applied={T|T < 7332556689455874048}]
I20260923 14:49:33.032750 16951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:33.032961 16951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:33.035861 16951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba1823b58b1d4a8294a24902476459e4: Sending DeleteTablet for 3 replicas of tablet 6278caf5e3174822a1c1b767d651c8a4
I20260923 14:49:33.036760 17339 tablet_service.cc:1514] Processing DeleteTablet for tablet 6278caf5e3174822a1c1b767d651c8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:33 UTC) from {username='slave'} at 127.0.0.1:38706
I20260923 14:49:33.036782 17075 tablet_service.cc:1514] Processing DeleteTablet for tablet 6278caf5e3174822a1c1b767d651c8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:33 UTC) from {username='slave'} at 127.0.0.1:46484
I20260923 14:49:33.036788 17207 tablet_service.cc:1514] Processing DeleteTablet for tablet 6278caf5e3174822a1c1b767d651c8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:33 UTC) from {username='slave'} at 127.0.0.1:51372
I20260923 14:49:33.037516 17446 tablet_replica.cc:330] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: stopping tablet replica
I20260923 14:49:33.037724 17447 tablet_replica.cc:330] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405: stopping tablet replica
I20260923 14:49:33.038053 17447 raft_consensus.cc:2229] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:49:33.038043 17446 raft_consensus.cc:2229] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:49:33.038396 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 17013
I20260923 14:49:33.038430 17447 raft_consensus.cc:2258] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:33.037364 17445 tablet_replica.cc:330] T 6278caf5e3174822a1c1b767d651c8a4 P 3dc282a2582a46c296fe5e3702896b0d: stopping tablet replica
I20260923 14:49:33.040972 17446 raft_consensus.cc:2258] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:33.041819 17447 ts_tablet_manager.cc:1905] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:49:33.042434 17446 ts_tablet_manager.cc:1905] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:49:33.045399 17446 ts_tablet_manager.cc:1918] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:49:33.045528 17446 log.cc:1196] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/6278caf5e3174822a1c1b767d651c8a4
I20260923 14:49:33.045581 17447 ts_tablet_manager.cc:1918] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:49:33.045686 17447 log.cc:1196] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/6278caf5e3174822a1c1b767d651c8a4
I20260923 14:49:33.045979 17446 ts_tablet_manager.cc:1939] T 6278caf5e3174822a1c1b767d651c8a4 P 170d7c1a8cc24e2e9003ddc91a2706bf: Deleting consensus metadata
I20260923 14:49:33.045994 17447 ts_tablet_manager.cc:1939] T 6278caf5e3174822a1c1b767d651c8a4 P 4b8a14b89acb46db87a6c9a435992405: Deleting consensus metadata
I20260923 14:49:33.047477 16940 catalog_manager.cc:4841] TS 4b8a14b89acb46db87a6c9a435992405 (127.30.2.195:33275): tablet 6278caf5e3174822a1c1b767d651c8a4 (table test-table [id=c903cf748c1f4d7ab4dfc743ea571af4]) successfully deleted
W20260923 14:49:33.047842 16938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:40807 (error 108)
I20260923 14:49:33.047909 16937 catalog_manager.cc:4841] TS 170d7c1a8cc24e2e9003ddc91a2706bf (127.30.2.194:39523): tablet 6278caf5e3174822a1c1b767d651c8a4 (table test-table [id=c903cf748c1f4d7ab4dfc743ea571af4]) successfully deleted
I20260923 14:49:33.047981 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 17144
I20260923 14:49:33.049386 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:33.048989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:40807 (local address 127.0.0.1:46500)
0923 14:49:33.049166 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:33.049276 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 14:49:33.049540 16938 catalog_manager.cc:4584] TS 3dc282a2582a46c296fe5e3702896b0d (127.30.2.193:40807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6278caf5e3174822a1c1b767d651c8a4: Network error: Client connection negotiation failed: client connection to 127.30.2.193:40807: connect: Connection refused (error 111)
I20260923 14:49:33.055991 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 17277
I20260923 14:49:33.062209 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 16922
2026-09-23T14:49:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1066 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:33 dist-test-slave-wjf2 krb5kdc[17451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:33 dist-test-slave-wjf2 krb5kdc[17451](info): set up 2 sockets
Sep 23 14:49:33 dist-test-slave-wjf2 krb5kdc[17451](info): commencing operation
krb5kdc: starting...
W20260923 14:49:34.109331 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:34 dist-test-slave-wjf2 krb5kdc[17451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:49:34.233047 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:37825
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:37825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:34.344818 17464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:34.345075 17464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:34.345115 17464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:34.349802 17464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:34.349857 17464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:34.349876 17464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:34.349895 17464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:34.349910 17464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:34.355136 17464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:37825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:37825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:34.356213 17464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:34.357316 17464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:34.367053 17470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:34.367208 17464 server_base.cc:1034] running on GCE node
W20260923 14:49:34.367053 17469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:34.367429 17472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:34.368276 17464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:34.369805 17464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:34.371006 17464 hybrid_clock.cc:648] HybridClock initialized: now 1790174974370976 us; error 36 us; skew 500 ppm
Sep 23 14:49:34 dist-test-slave-wjf2 krb5kdc[17451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:34.374528 17464 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:49:34.375429 17464 webserver.cc:460] Webserver started at http://127.30.2.254:35663/ using document root <none> and password file <none>
I20260923 14:49:34.376081 17464 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:34.376150 17464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:34.376400 17464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:34.378508 17464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "f2e77bef6b5241629474bd3300e1aab9"
format_stamp: "Formatted at 2026-09-23 14:49:34 on dist-test-slave-wjf2"
server_key: "bd1455b27c5f969aa357e415267165f1"
server_key_iv: "9474a4ac7cdb22042b2dd7e3a2f51c83"
server_key_version: "encryptionkey@0"
I20260923 14:49:34.379112 17464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "f2e77bef6b5241629474bd3300e1aab9"
format_stamp: "Formatted at 2026-09-23 14:49:34 on dist-test-slave-wjf2"
server_key: "bd1455b27c5f969aa357e415267165f1"
server_key_iv: "9474a4ac7cdb22042b2dd7e3a2f51c83"
server_key_version: "encryptionkey@0"
I20260923 14:49:34.383093 17464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 14:49:34.385694 17479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:34.387015 17464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:49:34.387164 17464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "f2e77bef6b5241629474bd3300e1aab9"
format_stamp: "Formatted at 2026-09-23 14:49:34 on dist-test-slave-wjf2"
server_key: "bd1455b27c5f969aa357e415267165f1"
server_key_iv: "9474a4ac7cdb22042b2dd7e3a2f51c83"
server_key_version: "encryptionkey@0"
I20260923 14:49:34.387284 17464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:34.411216 17464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:34.414434 17464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:34.414644 17464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:34.421933 17464 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:37825
I20260923 14:49:34.421963 17531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:37825 every 8 connection(s)
I20260923 14:49:34.422869 17464 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:34.425583 17532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:34.429467 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 17464
I20260923 14:49:34.429632 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:34.429915 30731 external_mini_cluster.cc:1428] Setting key 973e7f985675bcb0897dce3f0c5b4fdb
I20260923 14:49:34.431305 17532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9: Bootstrap starting.
I20260923 14:49:34.433564 17532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:34.434304 17532 log.cc:824] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:34.436154 17532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9: No bootstrap required, opened a new log
Sep 23 14:49:34 dist-test-slave-wjf2 krb5kdc[17451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:49:34.439559 17532 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e77bef6b5241629474bd3300e1aab9" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37825 } }
I20260923 14:49:34.439878 17532 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:34.439960 17532 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e77bef6b5241629474bd3300e1aab9, State: Initialized, Role: FOLLOWER
I20260923 14:49:34.440435 17532 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e77bef6b5241629474bd3300e1aab9" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37825 } }
I20260923 14:49:34.440585 17532 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:34.440665 17532 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:34.440788 17532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:34.441947 17532 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e77bef6b5241629474bd3300e1aab9" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37825 } }
I20260923 14:49:34.442339 17532 leader_election.cc:302] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2e77bef6b5241629474bd3300e1aab9; no voters:
I20260923 14:49:34.442714 17532 leader_election.cc:288] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:34.442787 17537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:34.442968 17537 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [term 1 LEADER]: Becoming Leader. State: Replica: f2e77bef6b5241629474bd3300e1aab9, State: Running, Role: LEADER
I20260923 14:49:34.443276 17537 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e77bef6b5241629474bd3300e1aab9" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37825 } }
I20260923 14:49:34.444059 17532 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:34.444881 17538 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2e77bef6b5241629474bd3300e1aab9. Latest consensus state: current_term: 1 leader_uuid: "f2e77bef6b5241629474bd3300e1aab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e77bef6b5241629474bd3300e1aab9" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37825 } } }
I20260923 14:49:34.445024 17538 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:34.445389 17539 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2e77bef6b5241629474bd3300e1aab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e77bef6b5241629474bd3300e1aab9" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37825 } } }
I20260923 14:49:34.445484 17539 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:34.445703 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:34.431378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.30.2.254:37825)
0923 14:49:34.431892 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:34.431909 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:34.431946 (+ 37us) server_negotiation.cc:407] Connection header received
0923 14:49:34.432597 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:34.432615 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:34.432953 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:34.433294 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:34.434164 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:34.435204 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:34.436174 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:34.436701 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:34.436901 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:34.439491 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:34.439497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:34.439511 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:34.439538 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:34.441696 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:34.442214 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:34.442220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:34.442234 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:34.442363 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:34.442706 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:34.442709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:34.442710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:34.443317 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:34.444190 (+ 873us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:34.444380 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 14:49:34.444678 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260923 14:49:34.445765 17546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:34.449575 17546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:34.455345 17546 catalog_manager.cc:1347] Generated new cluster ID: 7dd105d31e6b4459be29c028199128c0
I20260923 14:49:34.455421 17546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:34.467165 17546 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:34.468103 17546 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:34.479137 17546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9: Generated new TSK 0
I20260923 14:49:34.479990 17546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
Entry for principal kudu/127.30.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab.
I20260923 14:49:34.524214 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:37825
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:34.640900 17558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:34.641161 17558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:34.641228 17558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:34.645464 17558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:34.645539 17558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:34.645643 17558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:49:34.650929 17558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.193.keytab
--principal=kudu/127.30.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:34.651965 17558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:34.652977 17558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:34.662878 17563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:34.662853 17566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:34.663087 17558 server_base.cc:1034] running on GCE node
W20260923 14:49:34.662859 17564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:34.663600 17558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:34.664227 17558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:34.665442 17558 hybrid_clock.cc:648] HybridClock initialized: now 1790174974665426 us; error 30 us; skew 500 ppm
Sep 23 14:49:34 dist-test-slave-wjf2 krb5kdc[17451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:34.668453 17558 init.cc:377] Logged in from keytab as kudu/127.30.2.193@KRBTEST.COM (short username kudu)
I20260923 14:49:34.669297 17558 webserver.cc:460] Webserver started at http://127.30.2.193:32951/ using document root <none> and password file <none>
I20260923 14:49:34.669867 17558 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:34.669960 17558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:34.670199 17558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:34.672324 17558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "0ba88c72b6164ba8862e6140d22915b6"
format_stamp: "Formatted at 2026-09-23 14:49:34 on dist-test-slave-wjf2"
server_key: "f9d6e1bb49a2e8895c29e18986548a30"
server_key_iv: "6d41469ea93c63c8c66f580ac45a536b"
server_key_version: "encryptionkey@0"
I20260923 14:49:34.672902 17558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "0ba88c72b6164ba8862e6140d22915b6"
format_stamp: "Formatted at 2026-09-23 14:49:34 on dist-test-slave-wjf2"
server_key: "f9d6e1bb49a2e8895c29e18986548a30"
server_key_iv: "6d41469ea93c63c8c66f580ac45a536b"
server_key_version: "encryptionkey@0"
I20260923 14:49:34.676649 17558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:49:34.679180 17573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:34.680640 17558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:49:34.680764 17558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "0ba88c72b6164ba8862e6140d22915b6"
format_stamp: "Formatted at 2026-09-23 14:49:34 on dist-test-slave-wjf2"
server_key: "f9d6e1bb49a2e8895c29e18986548a30"
server_key_iv: "6d41469ea93c63c8c66f580ac45a536b"
server_key_version: "encryptionkey@0"
I20260923 14:49:34.680859 17558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:34.692376 17558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:34.694964 17558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:34.695102 17558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:34.695600 17558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:34.696352 17558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:34.696396 17558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:34.696434 17558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:34.696457 17558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:34.705185 17558 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:44399
I20260923 14:49:34.705209 17686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:44399 every 8 connection(s)
I20260923 14:49:34.706055 17558 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:49:34.711277 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 17558
I20260923 14:49:34.711429 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:49:34.711797 30731 external_mini_cluster.cc:1428] Setting key d3fccb916388c2a37603cba3ac7ea01a
Sep 23 14:49:34 dist-test-slave-wjf2 krb5kdc[17451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.193@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:49:34.720667 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:34.708012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:51559 (local address 127.30.2.254:37825)
0923 14:49:34.708179 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:34.708185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:34.709316 (+ 1131us) server_negotiation.cc:407] Connection header received
0923 14:49:34.710209 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:34.710212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:34.710267 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:34.710344 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:34.711518 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:34.712317 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:34.713635 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:34.713868 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:34.713951 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:34.716797 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:34.716800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:34.716802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:34.716825 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:34.718477 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:34.719068 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:34.719072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:34.719074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:34.719155 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:34.719518 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:34.719523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:34.719524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:34.719717 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:34.719855 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:34.720369 (+ 514us) server_negotiation.cc:299] Negotiation successful
0923 14:49:34.720497 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.30.2.194@KRBTEST.COM; defaulting to no policy
I20260923 14:49:34.721527 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:34.708493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:37825 (local address 127.30.2.193:51559)
0923 14:49:34.709160 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:34.709200 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:34.709973 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:34.710508 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:34.710514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:34.710847 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:34.711337 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:34.711354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:34.712461 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:34.712465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:34.713508 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:34.713521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:34.714035 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:34.714042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:34.714293 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:34.715070 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:49:34.715095 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:34.716608 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:34.718614 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:34.718621 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:34.718637 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:34.718957 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:34.719252 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:34.719255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:34.719257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:34.719417 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:34.719852 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:34.719860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:34.720124 (+ 264us) client_negotiation.cc:769] Sending connection context
0923 14:49:34.720318 (+ 194us) client_negotiation.cc:240] Negotiation successful
0923 14:49:34.720660 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":128,"threads_started":1}
I20260923 14:49:34.723016 17687 heartbeater.cc:347] Connected to a master server at 127.30.2.254:37825
I20260923 14:49:34.723312 17687 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:34.723965 17687 heartbeater.cc:511] Master 127.30.2.254:37825 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.194@KRBTEST.COM" created.
I20260923 14:49:34.725762 17496 ts_manager.cc:194] Registered new tserver with Master: 0ba88c72b6164ba8862e6140d22915b6 (127.30.2.193:44399)
I20260923 14:49:34.727326 17496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.193@KRBTEST.COM'} at 127.30.2.193:51559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
Entry for principal kudu/127.30.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab.
I20260923 14:49:34.742436 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:37825
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:34.853540 17692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:34.853778 17692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:34.853824 17692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:34.857733 17692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:34.857776 17692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:34.857831 17692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:49:34.863000 17692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.194.keytab
--principal=kudu/127.30.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:34.863950 17692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:34.864867 17692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:34.874058 17697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:34.874064 17698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:34.874065 17700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:34.874320 17692 server_base.cc:1034] running on GCE node
I20260923 14:49:34.874794 17692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:34.875403 17692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:34.876565 17692 hybrid_clock.cc:648] HybridClock initialized: now 1790174974876545 us; error 29 us; skew 500 ppm
Sep 23 14:49:34 dist-test-slave-wjf2 krb5kdc[17451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:34.879367 17692 init.cc:377] Logged in from keytab as kudu/127.30.2.194@KRBTEST.COM (short username kudu)
I20260923 14:49:34.880288 17692 webserver.cc:460] Webserver started at http://127.30.2.194:43819/ using document root <none> and password file <none>
I20260923 14:49:34.880856 17692 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:34.880914 17692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:34.881115 17692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:34.883042 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "1e03ad625ce94d3097666f481e652ac2"
format_stamp: "Formatted at 2026-09-23 14:49:34 on dist-test-slave-wjf2"
server_key: "d06f458264b59f3264e60ea96c130400"
server_key_iv: "0f64d3febc0518d2c31f263fe94f716b"
server_key_version: "encryptionkey@0"
I20260923 14:49:34.883635 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "1e03ad625ce94d3097666f481e652ac2"
format_stamp: "Formatted at 2026-09-23 14:49:34 on dist-test-slave-wjf2"
server_key: "d06f458264b59f3264e60ea96c130400"
server_key_iv: "0f64d3febc0518d2c31f263fe94f716b"
server_key_version: "encryptionkey@0"
I20260923 14:49:34.887296 17692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:49:34.889799 17707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:34.890995 17692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:49:34.891140 17692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "1e03ad625ce94d3097666f481e652ac2"
format_stamp: "Formatted at 2026-09-23 14:49:34 on dist-test-slave-wjf2"
server_key: "d06f458264b59f3264e60ea96c130400"
server_key_iv: "0f64d3febc0518d2c31f263fe94f716b"
server_key_version: "encryptionkey@0"
I20260923 14:49:34.891243 17692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:34.913578 17692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:34.917021 17692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:34.917229 17692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:34.917869 17692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:34.918656 17692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:34.918728 17692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:34.918802 17692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:34.918840 17692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:34.928670 17692 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:43891
I20260923 14:49:34.928782 17820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:43891 every 8 connection(s)
I20260923 14:49:34.929585 17692 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:49:34.939200 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 17692
I20260923 14:49:34.939337 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
Sep 23 14:49:34 dist-test-slave-wjf2 krb5kdc[17451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.194@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:49:34.939625 30731 external_mini_cluster.cc:1428] Setting key fa456fa84e9fb5184ecc248346392e2a
I20260923 14:49:34.943907 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:34.931468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:33481 (local address 127.30.2.254:37825)
0923 14:49:34.931620 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:34.931624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:34.932990 (+ 1366us) server_negotiation.cc:407] Connection header received
0923 14:49:34.933939 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:34.933944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:34.934007 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:34.934119 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:34.935312 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:34.935917 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:34.937051 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:34.937307 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:34.937436 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:34.940067 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:34.940071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:34.940074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:34.940107 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:34.941610 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:34.942201 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:34.942207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:34.942209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:34.942305 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:34.942684 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:34.942688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:34.942690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:34.942902 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:34.943018 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:34.943541 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 14:49:34.943669 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:49:34.944892 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:34.931868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:37825 (local address 127.30.2.194:33481)
0923 14:49:34.932819 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:34.932864 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:34.933708 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:34.934312 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:34.934320 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:34.934677 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:34.935146 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:34.935160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:34.936049 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:34.936055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:34.936913 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:34.936923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:34.937471 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:34.937476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:34.937671 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:34.938224 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:49:34.938243 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:34.939895 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:34.941759 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:34.941765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:34.941779 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:34.942087 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:34.942417 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:34.942420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:34.942423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:34.942572 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:34.943031 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:34.943040 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:34.943283 (+ 243us) client_negotiation.cc:769] Sending connection context
0923 14:49:34.943477 (+ 194us) client_negotiation.cc:240] Negotiation successful
0923 14:49:34.943866 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":535,"threads_started":1}
I20260923 14:49:34.946092 17821 heartbeater.cc:347] Connected to a master server at 127.30.2.254:37825
I20260923 14:49:34.946352 17821 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:34.946986 17821 heartbeater.cc:511] Master 127.30.2.254:37825 requested a full tablet report, sending...
I20260923 14:49:34.948083 17496 ts_manager.cc:194] Registered new tserver with Master: 1e03ad625ce94d3097666f481e652ac2 (127.30.2.194:43891)
I20260923 14:49:34.948629 17496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.194@KRBTEST.COM'} at 127.30.2.194:33481
WARNING: no policy specified for kudu/127.30.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
Entry for principal kudu/127.30.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab.
I20260923 14:49:34.966754 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:37825
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:35.073477 17826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:35.073710 17826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:35.073750 17826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:35.077704 17826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:35.077751 17826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:35.077818 17826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:49:35.082887 17826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.195.keytab
--principal=kudu/127.30.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:35.083853 17826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:35.084909 17826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:35.094504 17832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:35.094504 17834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:35.094516 17831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:35.095273 17826 server_base.cc:1034] running on GCE node
I20260923 14:49:35.095709 17826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:35.096436 17826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:35.097648 17826 hybrid_clock.cc:648] HybridClock initialized: now 1790174975097603 us; error 57 us; skew 500 ppm
Sep 23 14:49:35 dist-test-slave-wjf2 krb5kdc[17451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174975, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:35.100911 17826 init.cc:377] Logged in from keytab as kudu/127.30.2.195@KRBTEST.COM (short username kudu)
I20260923 14:49:35.101806 17826 webserver.cc:460] Webserver started at http://127.30.2.195:40491/ using document root <none> and password file <none>
I20260923 14:49:35.102392 17826 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:35.102459 17826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:35.102641 17826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:35.104892 17826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "df5a43495958414caff398cef2a57fbd"
format_stamp: "Formatted at 2026-09-23 14:49:35 on dist-test-slave-wjf2"
server_key: "9a51f05e640bd5c64cebc60b66bce865"
server_key_iv: "1d761a077f7c1b5ba7090fed7f34f70e"
server_key_version: "encryptionkey@0"
I20260923 14:49:35.105479 17826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "df5a43495958414caff398cef2a57fbd"
format_stamp: "Formatted at 2026-09-23 14:49:35 on dist-test-slave-wjf2"
server_key: "9a51f05e640bd5c64cebc60b66bce865"
server_key_iv: "1d761a077f7c1b5ba7090fed7f34f70e"
server_key_version: "encryptionkey@0"
I20260923 14:49:35.109579 17826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:49:35.112308 17841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:35.113586 17826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:35.113684 17826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "df5a43495958414caff398cef2a57fbd"
format_stamp: "Formatted at 2026-09-23 14:49:35 on dist-test-slave-wjf2"
server_key: "9a51f05e640bd5c64cebc60b66bce865"
server_key_iv: "1d761a077f7c1b5ba7090fed7f34f70e"
server_key_version: "encryptionkey@0"
I20260923 14:49:35.113775 17826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:35.123481 17826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:35.126714 17826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:35.126859 17826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:35.127346 17826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:35.128103 17826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:35.128189 17826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:35.128247 17826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:35.128284 17826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:35.137374 17826 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:45759
I20260923 14:49:35.137396 17954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:45759 every 8 connection(s)
I20260923 14:49:35.138213 17826 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:49:35.142205 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 17826
I20260923 14:49:35.142324 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:49:35.142637 30731 external_mini_cluster.cc:1428] Setting key b07bda744e21ffec66c1ec214c96c24f
Sep 23 14:49:35 dist-test-slave-wjf2 krb5kdc[17451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174975, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.195@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:49:35.153301 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.140004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:57201 (local address 127.30.2.254:37825)
0923 14:49:35.140140 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:35.140144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:35.141167 (+ 1023us) server_negotiation.cc:407] Connection header received
0923 14:49:35.142050 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:35.142053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:35.142100 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:35.142189 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:35.143646 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.144463 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.146120 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.146374 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.146527 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.149418 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:35.149431 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:35.149435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:35.149462 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:35.150890 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:35.151470 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:35.151476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:35.151478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:35.151578 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:35.152073 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:35.152077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:35.152079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:35.152298 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:35.152426 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:35.152964 (+ 538us) server_negotiation.cc:299] Negotiation successful
0923 14:49:35.153094 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:49:35.154052 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.140371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:37825 (local address 127.30.2.195:57201)
0923 14:49:35.141021 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:35.141058 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:35.141803 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:35.142415 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:35.142425 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:35.142908 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:35.143460 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.143479 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.144617 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.144621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.145968 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.145980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.146560 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.146566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.146790 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.147538 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:49:35.147557 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:35.149229 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:35.151028 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:35.151035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:35.151060 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:35.151346 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:35.151709 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:49:35.151713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:49:35.151715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:49:35.151952 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:49:35.152423 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:35.152431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:35.152685 (+ 254us) client_negotiation.cc:769] Sending connection context
0923 14:49:35.152895 (+ 210us) client_negotiation.cc:240] Negotiation successful
0923 14:49:35.153219 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":108,"threads_started":1}
I20260923 14:49:35.155289 17955 heartbeater.cc:347] Connected to a master server at 127.30.2.254:37825
I20260923 14:49:35.155577 17955 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:35.156181 17955 heartbeater.cc:511] Master 127.30.2.254:37825 requested a full tablet report, sending...
I20260923 14:49:35.157308 17496 ts_manager.cc:194] Registered new tserver with Master: df5a43495958414caff398cef2a57fbd (127.30.2.195:45759)
I20260923 14:49:35.157882 17496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.2.195@KRBTEST.COM'} at 127.30.2.195:57201
I20260923 14:49:35.168370 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:49:35.177686 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.170350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.30.2.254:37825)
0923 14:49:35.170487 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:35.170492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:35.170684 (+ 192us) server_negotiation.cc:407] Connection header received
0923 14:49:35.170840 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:35.170843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:35.170906 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:35.171012 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:35.171707 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.172254 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.173079 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.173293 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.173400 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.174680 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:35.174684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:35.174687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:35.174712 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:35.175978 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:35.176404 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:35.176412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:35.176413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:35.176485 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:35.176763 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:35.176766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:35.176768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:35.177020 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:35.177164 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:35.177413 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 14:49:35.177531 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:49:35.180260 17496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:49:35.183230 17496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:49:35.197628 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.192288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.30.2.195:45759)
0923 14:49:35.193036 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:35.193048 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:35.193072 (+ 24us) server_negotiation.cc:407] Connection header received
0923 14:49:35.193151 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:35.193157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:35.193368 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:35.193579 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:35.193874 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.194850 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.196427 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.196968 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.197089 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.197238 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:35.197394 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 14:49:35.197482 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":134,"threads_started":1}
I20260923 14:49:35.197616 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.191910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:45759 (local address 127.0.0.1:49374)
0923 14:49:35.192267 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:35.192313 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:35.192431 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:35.193580 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:35.193587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:35.193610 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:35.193756 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.193778 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.194979 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.194985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.196297 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.196306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.197090 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.197093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.197206 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.197243 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 14:49:35.197364 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 14:49:35.197479 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":114,"threads_started":1}
I20260923 14:49:35.198845 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.192661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:44399 (local address 127.0.0.1:44572)
0923 14:49:35.193650 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:35.193662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:35.193742 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:35.194210 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:35.194213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:35.194234 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:35.194318 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.194323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.195808 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.195811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.197121 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.197130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.198435 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.198439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.198537 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.198562 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 14:49:35.198628 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 14:49:35.198705 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":155,"thread_start_us":100,"threads_started":1}
I20260923 14:49:35.199040 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.192804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.30.2.194:43891)
0923 14:49:35.193224 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:35.193241 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:35.193477 (+ 236us) server_negotiation.cc:407] Connection header received
0923 14:49:35.193581 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:35.193586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:35.193771 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:35.193912 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:35.194397 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.195628 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.197243 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.198076 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.198183 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.198310 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:35.198789 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 14:49:35.198884 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":89,"threads_started":1}
I20260923 14:49:35.199301 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.191951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:43891 (local address 127.0.0.1:52298)
0923 14:49:35.193375 (+ 1424us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:35.193385 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:35.193462 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:35.194083 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:35.194086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:35.194105 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:35.194220 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.194226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.195808 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.195811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.197088 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.197101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.198379 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.198385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.198482 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.198505 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 14:49:35.198573 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 14:49:35.198644 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1361,"mutex_wait_us":31,"thread_start_us":95,"threads_started":1}
I20260923 14:49:35.199393 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.192815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.30.2.193:44399)
0923 14:49:35.193227 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:35.193240 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:35.193752 (+ 512us) server_negotiation.cc:407] Connection header received
0923 14:49:35.193839 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:35.193847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:35.194083 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:35.194235 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:35.194407 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.195636 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.197243 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.198312 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.198421 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.198568 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:35.199148 (+ 580us) server_negotiation.cc:299] Negotiation successful
0923 14:49:35.199243 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":135,"threads_started":1}
I20260923 14:49:35.200877 17889 tablet_service.cc:1467] Processing CreateTablet for tablet 4efed037ac5f45b99a5049e4af178d66 (DEFAULT_TABLE table=test-table [id=487cce8b5f1d49fa962b992133c35195]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:35.202019 17889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efed037ac5f45b99a5049e4af178d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:35.201781 17621 tablet_service.cc:1467] Processing CreateTablet for tablet 4efed037ac5f45b99a5049e4af178d66 (DEFAULT_TABLE table=test-table [id=487cce8b5f1d49fa962b992133c35195]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:35.202608 17621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efed037ac5f45b99a5049e4af178d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:35.202683 17755 tablet_service.cc:1467] Processing CreateTablet for tablet 4efed037ac5f45b99a5049e4af178d66 (DEFAULT_TABLE table=test-table [id=487cce8b5f1d49fa962b992133c35195]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:35.203588 17755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efed037ac5f45b99a5049e4af178d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:35.208498 17972 tablet_bootstrap.cc:492] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6: Bootstrap starting.
I20260923 14:49:35.210183 17973 tablet_bootstrap.cc:492] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: Bootstrap starting.
I20260923 14:49:35.210829 17972 tablet_bootstrap.cc:654] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:35.211958 17972 log.cc:824] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:35.212133 17974 tablet_bootstrap.cc:492] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: Bootstrap starting.
I20260923 14:49:35.212276 17973 tablet_bootstrap.cc:654] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:35.213495 17973 log.cc:824] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:35.214042 17972 tablet_bootstrap.cc:492] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6: No bootstrap required, opened a new log
I20260923 14:49:35.214254 17972 ts_tablet_manager.cc:1397] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260923 14:49:35.214757 17974 tablet_bootstrap.cc:654] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:35.215547 17974 log.cc:824] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:35.215598 17973 tablet_bootstrap.cc:492] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: No bootstrap required, opened a new log
I20260923 14:49:35.215765 17973 ts_tablet_manager.cc:1397] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:49:35.217489 17974 tablet_bootstrap.cc:492] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: No bootstrap required, opened a new log
I20260923 14:49:35.217672 17974 ts_tablet_manager.cc:1397] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:49:35.217653 17972 raft_consensus.cc:348] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a43495958414caff398cef2a57fbd" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45759 } } peers { permanent_uuid: "1e03ad625ce94d3097666f481e652ac2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 43891 } } peers { permanent_uuid: "0ba88c72b6164ba8862e6140d22915b6" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 44399 } }
I20260923 14:49:35.217947 17972 raft_consensus.cc:374] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:35.218009 17972 raft_consensus.cc:729] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba88c72b6164ba8862e6140d22915b6, State: Initialized, Role: FOLLOWER
I20260923 14:49:35.218456 17972 consensus_queue.cc:260] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a43495958414caff398cef2a57fbd" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45759 } } peers { permanent_uuid: "1e03ad625ce94d3097666f481e652ac2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 43891 } } peers { permanent_uuid: "0ba88c72b6164ba8862e6140d22915b6" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 44399 } }
I20260923 14:49:35.219110 17687 heartbeater.cc:503] Master 127.30.2.254:37825 was elected leader, sending a full tablet report...
I20260923 14:49:35.219478 17972 ts_tablet_manager.cc:1428] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:49:35.219607 17973 raft_consensus.cc:348] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a43495958414caff398cef2a57fbd" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45759 } } peers { permanent_uuid: "1e03ad625ce94d3097666f481e652ac2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 43891 } } peers { permanent_uuid: "0ba88c72b6164ba8862e6140d22915b6" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 44399 } }
I20260923 14:49:35.219871 17973 raft_consensus.cc:374] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:35.219918 17973 raft_consensus.cc:729] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5a43495958414caff398cef2a57fbd, State: Initialized, Role: FOLLOWER
I20260923 14:49:35.220310 17973 consensus_queue.cc:260] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a43495958414caff398cef2a57fbd" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45759 } } peers { permanent_uuid: "1e03ad625ce94d3097666f481e652ac2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 43891 } } peers { permanent_uuid: "0ba88c72b6164ba8862e6140d22915b6" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 44399 } }
I20260923 14:49:35.221225 17973 ts_tablet_manager.cc:1428] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260923 14:49:35.221407 17974 raft_consensus.cc:348] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a43495958414caff398cef2a57fbd" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45759 } } peers { permanent_uuid: "1e03ad625ce94d3097666f481e652ac2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 43891 } } peers { permanent_uuid: "0ba88c72b6164ba8862e6140d22915b6" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 44399 } }
I20260923 14:49:35.221611 17955 heartbeater.cc:503] Master 127.30.2.254:37825 was elected leader, sending a full tablet report...
I20260923 14:49:35.221700 17974 raft_consensus.cc:374] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:35.221761 17974 raft_consensus.cc:729] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e03ad625ce94d3097666f481e652ac2, State: Initialized, Role: FOLLOWER
I20260923 14:49:35.222151 17974 consensus_queue.cc:260] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a43495958414caff398cef2a57fbd" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45759 } } peers { permanent_uuid: "1e03ad625ce94d3097666f481e652ac2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 43891 } } peers { permanent_uuid: "0ba88c72b6164ba8862e6140d22915b6" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 44399 } }
I20260923 14:49:35.223086 17974 ts_tablet_manager.cc:1428] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 14:49:35.223435 17821 heartbeater.cc:503] Master 127.30.2.254:37825 was elected leader, sending a full tablet report...
I20260923 14:49:35.373129 17980 raft_consensus.cc:482] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:49:35.373379 17980 raft_consensus.cc:504] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a43495958414caff398cef2a57fbd" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45759 } } peers { permanent_uuid: "1e03ad625ce94d3097666f481e652ac2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 43891 } } peers { permanent_uuid: "0ba88c72b6164ba8862e6140d22915b6" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 44399 } }
I20260923 14:49:35.374679 17980 leader_election.cc:288] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df5a43495958414caff398cef2a57fbd (127.30.2.195:45759), 0ba88c72b6164ba8862e6140d22915b6 (127.30.2.193:44399)
I20260923 14:49:35.378381 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.374905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:45759 (local address 127.30.2.194:45587)
0923 14:49:35.375030 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:35.375049 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:35.375151 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:35.375468 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:35.375471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:35.375490 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:35.375614 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.375619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.376431 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.376436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.377283 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.377292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.377935 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.377938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.378046 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.378075 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 14:49:35.378165 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 14:49:35.378235 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 14:49:35.378381 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.375034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:45587 (local address 127.30.2.195:45759)
0923 14:49:35.375188 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:35.375193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:35.375209 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:49:35.375297 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:35.375302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:35.375351 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:35.375454 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:35.375737 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.376317 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.377416 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.377815 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.377943 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.377997 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:35.378184 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 14:49:35.378248 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:49:35.378610 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.375218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:49607 (local address 127.30.2.193:44399)
0923 14:49:35.375348 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:35.375353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:35.375512 (+ 159us) server_negotiation.cc:407] Connection header received
0923 14:49:35.375665 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:35.375668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:35.375711 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:35.375820 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:35.376092 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.376623 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.377619 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.378006 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.378138 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.378192 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:35.378388 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 14:49:35.378469 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:49:35.378945 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.375003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:44399 (local address 127.30.2.194:49607)
0923 14:49:35.375425 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:35.375435 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:35.375549 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:35.375842 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:35.375845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:35.375861 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:35.375982 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.375987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.376732 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.376736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.377493 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.377502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.378130 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.378134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.378238 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.378259 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 14:49:35.378739 (+ 480us) client_negotiation.cc:240] Negotiation successful
0923 14:49:35.378821 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":136,"threads_started":1}
I20260923 14:49:35.379078 17909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efed037ac5f45b99a5049e4af178d66" candidate_uuid: "1e03ad625ce94d3097666f481e652ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a43495958414caff398cef2a57fbd" is_pre_election: true
I20260923 14:49:35.379410 17909 raft_consensus.cc:2454] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e03ad625ce94d3097666f481e652ac2 in term 0.
I20260923 14:49:35.379535 17641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efed037ac5f45b99a5049e4af178d66" candidate_uuid: "1e03ad625ce94d3097666f481e652ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba88c72b6164ba8862e6140d22915b6" is_pre_election: true
I20260923 14:49:35.379841 17641 raft_consensus.cc:2454] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e03ad625ce94d3097666f481e652ac2 in term 0.
I20260923 14:49:35.379944 17709 leader_election.cc:302] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e03ad625ce94d3097666f481e652ac2, df5a43495958414caff398cef2a57fbd; no voters:
I20260923 14:49:35.380226 17980 raft_consensus.cc:2781] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:49:35.380319 17980 raft_consensus.cc:482] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:49:35.380362 17980 raft_consensus.cc:3037] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:35.381371 17980 raft_consensus.cc:504] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a43495958414caff398cef2a57fbd" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45759 } } peers { permanent_uuid: "1e03ad625ce94d3097666f481e652ac2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 43891 } } peers { permanent_uuid: "0ba88c72b6164ba8862e6140d22915b6" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 44399 } }
I20260923 14:49:35.381706 17980 leader_election.cc:288] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [CANDIDATE]: Term 1 election: Requested vote from peers df5a43495958414caff398cef2a57fbd (127.30.2.195:45759), 0ba88c72b6164ba8862e6140d22915b6 (127.30.2.193:44399)
I20260923 14:49:35.382014 17909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efed037ac5f45b99a5049e4af178d66" candidate_uuid: "1e03ad625ce94d3097666f481e652ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a43495958414caff398cef2a57fbd"
I20260923 14:49:35.382114 17641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efed037ac5f45b99a5049e4af178d66" candidate_uuid: "1e03ad625ce94d3097666f481e652ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba88c72b6164ba8862e6140d22915b6"
I20260923 14:49:35.382165 17909 raft_consensus.cc:3037] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:35.382251 17641 raft_consensus.cc:3037] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:35.383203 17909 raft_consensus.cc:2454] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e03ad625ce94d3097666f481e652ac2 in term 1.
I20260923 14:49:35.383256 17641 raft_consensus.cc:2454] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e03ad625ce94d3097666f481e652ac2 in term 1.
I20260923 14:49:35.383510 17709 leader_election.cc:302] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e03ad625ce94d3097666f481e652ac2, df5a43495958414caff398cef2a57fbd; no voters:
I20260923 14:49:35.383689 17980 raft_consensus.cc:2781] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:35.383967 17980 raft_consensus.cc:686] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 1e03ad625ce94d3097666f481e652ac2, State: Running, Role: LEADER
I20260923 14:49:35.384296 17980 consensus_queue.cc:237] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a43495958414caff398cef2a57fbd" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45759 } } peers { permanent_uuid: "1e03ad625ce94d3097666f481e652ac2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 43891 } } peers { permanent_uuid: "0ba88c72b6164ba8862e6140d22915b6" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 44399 } }
I20260923 14:49:35.387637 17495 catalog_manager.cc:5495] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e03ad625ce94d3097666f481e652ac2 (127.30.2.194). New cstate: current_term: 1 leader_uuid: "1e03ad625ce94d3097666f481e652ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a43495958414caff398cef2a57fbd" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e03ad625ce94d3097666f481e652ac2" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 43891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ba88c72b6164ba8862e6140d22915b6" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 44399 } health_report { overall_health: UNKNOWN } } }
W20260923 14:49:35.389093 17956 tablet.cc:2369] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:49:35.430749 17822 tablet.cc:2369] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:49:35.434190 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.430340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.30.2.194:43891)
0923 14:49:35.430474 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:35.430478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:35.430492 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:49:35.430623 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:35.430625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:35.430687 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:35.430815 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:49:35.431092 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.431602 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.432591 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.432826 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.432953 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.433332 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:49:35.433720 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:49:35.433864 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:35.433945 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 14:49:35.434006 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:49:35.440173 17641 raft_consensus.cc:1261] T 4efed037ac5f45b99a5049e4af178d66 P 0ba88c72b6164ba8862e6140d22915b6 [term 1 FOLLOWER]: Refusing update from remote peer 1e03ad625ce94d3097666f481e652ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:35.440173 17909 raft_consensus.cc:1261] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd [term 1 FOLLOWER]: Refusing update from remote peer 1e03ad625ce94d3097666f481e652ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:35.440871 17982 consensus_queue.cc:1035] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba88c72b6164ba8862e6140d22915b6" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 44399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:35.441025 17980 consensus_queue.cc:1035] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5a43495958414caff398cef2a57fbd" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:35.450368 17987 mvcc.cc:205] Tried to move back new op lower bound from 7332556699395579904 to 7332556699176960000. Current Snapshot: MvccSnapshot[applied={T|T < 7332556699395579904}]
I20260923 14:49:35.455968 17495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:35.456121 17495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:35.458648 17495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2e77bef6b5241629474bd3300e1aab9: Sending DeleteTablet for 3 replicas of tablet 4efed037ac5f45b99a5049e4af178d66
I20260923 14:49:35.459416 17889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efed037ac5f45b99a5049e4af178d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:35 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:49374
I20260923 14:49:35.459530 17755 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efed037ac5f45b99a5049e4af178d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:35 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:52298
I20260923 14:49:35.459530 17621 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efed037ac5f45b99a5049e4af178d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:35 UTC) from {username='kudu', principal='kudu/127.30.2.254@KRBTEST.COM'} at 127.0.0.1:44572
I20260923 14:49:35.460170 17996 tablet_replica.cc:330] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: stopping tablet replica
I20260923 14:49:35.460233 17994 tablet_replica.cc:330] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: stopping tablet replica
I20260923 14:49:35.460474 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 17558
I20260923 14:49:35.460582 17996 raft_consensus.cc:2229] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:49:35.460676 17994 raft_consensus.cc:2229] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:49:35.460978 17996 raft_consensus.cc:2258] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:35.461040 17994 raft_consensus.cc:2258] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:35.462824 17996 ts_tablet_manager.cc:1905] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:49:35.464084 17994 ts_tablet_manager.cc:1905] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:49:35.466409 17996 ts_tablet_manager.cc:1918] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:49:35.466524 17996 log.cc:1196] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/4efed037ac5f45b99a5049e4af178d66
I20260923 14:49:35.466849 17996 ts_tablet_manager.cc:1939] T 4efed037ac5f45b99a5049e4af178d66 P 1e03ad625ce94d3097666f481e652ac2: Deleting consensus metadata
I20260923 14:49:35.466992 17994 ts_tablet_manager.cc:1918] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:49:35.467128 17994 log.cc:1196] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/4efed037ac5f45b99a5049e4af178d66
I20260923 14:49:35.467491 17994 ts_tablet_manager.cc:1939] T 4efed037ac5f45b99a5049e4af178d66 P df5a43495958414caff398cef2a57fbd: Deleting consensus metadata
I20260923 14:49:35.468076 17482 catalog_manager.cc:4841] TS 1e03ad625ce94d3097666f481e652ac2 (127.30.2.194:43891): tablet 4efed037ac5f45b99a5049e4af178d66 (table test-table [id=487cce8b5f1d49fa962b992133c35195]) successfully deleted
I20260923 14:49:35.468767 17481 catalog_manager.cc:4841] TS df5a43495958414caff398cef2a57fbd (127.30.2.195:45759): tablet 4efed037ac5f45b99a5049e4af178d66 (table test-table [id=487cce8b5f1d49fa962b992133c35195]) successfully deleted
W20260923 14:49:35.469710 17482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:44399 (error 108)
I20260923 14:49:35.470700 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.470401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:44399 (local address 127.0.0.1:44588)
0923 14:49:35.470526 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:35.470601 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:44399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 14:49:35.470827 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 17692
W20260923 14:49:35.470929 17482 catalog_manager.cc:4584] TS 0ba88c72b6164ba8862e6140d22915b6 (127.30.2.193:44399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4efed037ac5f45b99a5049e4af178d66: Network error: Client connection negotiation failed: client connection to 127.30.2.193:44399: connect: Connection refused (error 111)
I20260923 14:49:35.476897 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 17826
I20260923 14:49:35.482420 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 17464
2026-09-23T14:49:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2419 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T14:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:35Z Disabled control of system clock
I20260923 14:49:35.512763 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:42699
--webserver_interface=127.30.2.254
--webserver_port=0
--builtin_ntp_servers=127.30.2.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:42699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:49:35.626005 18001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:35.626255 18001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:35.626291 18001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:35.630416 18001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:35.630471 18001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:35.630491 18001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:35.630553 18001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:35.630589 18001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:35.635224 18001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:42699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:42699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:35.636317 18001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:35.637853 18001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:49:35.646554 18001 server_base.cc:1034] running on GCE node
W20260923 14:49:35.646541 18006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:35.646557 18009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:35.646544 18007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:35.647454 18001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:35.648707 18001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:35.649926 18001 hybrid_clock.cc:648] HybridClock initialized: now 1790174975649904 us; error 32 us; skew 500 ppm
I20260923 14:49:35.651706 18001 webserver.cc:460] Webserver started at http://127.30.2.254:43409/ using document root <none> and password file <none>
I20260923 14:49:35.652303 18001 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:35.652365 18001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:35.652592 18001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:35.654579 18001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "94d7570387d648f59e0e2b1660ba907a"
format_stamp: "Formatted at 2026-09-23 14:49:35 on dist-test-slave-wjf2"
server_key: "6acc5334efb231efa3f6c97402c29608"
server_key_iv: "0895d05291d58e1336a69a40630e7cd2"
server_key_version: "encryptionkey@0"
I20260923 14:49:35.655160 18001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "94d7570387d648f59e0e2b1660ba907a"
format_stamp: "Formatted at 2026-09-23 14:49:35 on dist-test-slave-wjf2"
server_key: "6acc5334efb231efa3f6c97402c29608"
server_key_iv: "0895d05291d58e1336a69a40630e7cd2"
server_key_version: "encryptionkey@0"
I20260923 14:49:35.658715 18001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 14:49:35.661176 18015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:35.662392 18001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:49:35.662540 18001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "94d7570387d648f59e0e2b1660ba907a"
format_stamp: "Formatted at 2026-09-23 14:49:35 on dist-test-slave-wjf2"
server_key: "6acc5334efb231efa3f6c97402c29608"
server_key_iv: "0895d05291d58e1336a69a40630e7cd2"
server_key_version: "encryptionkey@0"
I20260923 14:49:35.662648 18001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:35.688952 18001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:35.689613 18001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:35.689824 18001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:35.696678 18001 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:42699
I20260923 14:49:35.696695 18068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:42699 every 8 connection(s)
I20260923 14:49:35.697562 18001 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:35.698536 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 18001
I20260923 14:49:35.698673 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:35.699062 30731 external_mini_cluster.cc:1428] Setting key 40e6791ec5981bc589dce35e28e8bc22
I20260923 14:49:35.702008 18069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:35.708416 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.700583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.30.2.254:42699)
0923 14:49:35.701163 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:35.701176 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:35.701210 (+ 34us) server_negotiation.cc:407] Connection header received
0923 14:49:35.701836 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:35.701851 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:35.702281 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:35.702548 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:35.702943 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.703961 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.705171 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.705632 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.705746 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.706706 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:35.706711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:35.706723 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:35.707089 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:35.707141 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:35.707160 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:35.707234 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:35.707422 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 14:49:35.707695 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260923 14:49:35.708845 18069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a: Bootstrap starting.
I20260923 14:49:35.710933 18069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:35.711638 18069 log.cc:824] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:35.713469 18069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a: No bootstrap required, opened a new log
I20260923 14:49:35.716953 18069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7570387d648f59e0e2b1660ba907a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42699 } }
I20260923 14:49:35.717157 18069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:35.717232 18069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d7570387d648f59e0e2b1660ba907a, State: Initialized, Role: FOLLOWER
I20260923 14:49:35.717658 18069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7570387d648f59e0e2b1660ba907a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42699 } }
I20260923 14:49:35.717797 18069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:35.717890 18069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:35.718001 18069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:35.718856 18069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7570387d648f59e0e2b1660ba907a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42699 } }
I20260923 14:49:35.719179 18069 leader_election.cc:302] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94d7570387d648f59e0e2b1660ba907a; no voters:
I20260923 14:49:35.719475 18069 leader_election.cc:288] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:35.719673 18074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:35.719977 18074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [term 1 LEADER]: Becoming Leader. State: Replica: 94d7570387d648f59e0e2b1660ba907a, State: Running, Role: LEADER
I20260923 14:49:35.720304 18074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7570387d648f59e0e2b1660ba907a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42699 } }
I20260923 14:49:35.720829 18069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:35.721822 18075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94d7570387d648f59e0e2b1660ba907a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7570387d648f59e0e2b1660ba907a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42699 } } }
I20260923 14:49:35.721966 18075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:35.722378 18083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:35.722339 18076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94d7570387d648f59e0e2b1660ba907a. Latest consensus state: current_term: 1 leader_uuid: "94d7570387d648f59e0e2b1660ba907a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7570387d648f59e0e2b1660ba907a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42699 } } }
I20260923 14:49:35.722484 18076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:35.725775 18083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:35.731565 18083 catalog_manager.cc:1347] Generated new cluster ID: d369f0293f954b4c8d521308eac8d7e4
I20260923 14:49:35.731632 18083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:35.744169 18083 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:35.745397 18083 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:35.765758 18083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a: Generated new TSK 0
I20260923 14:49:35.766680 18083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:35.770231 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.193:0
--local_ip_for_outbound_sockets=127.30.2.193
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42699
--builtin_ntp_servers=127.30.2.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:49:35.885510 18093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:35.885807 18093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:35.885901 18093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:35.889891 18093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:35.889976 18093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:35.890075 18093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.193
I20260923 14:49:35.894698 18093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.2.193
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.193
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:35.895767 18093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:35.897225 18093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:35.907080 18101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:35.907042 18099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:35.907083 18098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:35.907344 18093 server_base.cc:1034] running on GCE node
I20260923 14:49:35.907866 18093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:35.908555 18093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:35.909744 18093 hybrid_clock.cc:648] HybridClock initialized: now 1790174975909720 us; error 36 us; skew 500 ppm
I20260923 14:49:35.911690 18093 webserver.cc:460] Webserver started at http://127.30.2.193:37703/ using document root <none> and password file <none>
I20260923 14:49:35.912335 18093 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:35.912432 18093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:35.912654 18093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:35.914706 18093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/data/instance:
uuid: "1b1433920de946a492c43ca178f9ad5c"
format_stamp: "Formatted at 2026-09-23 14:49:35 on dist-test-slave-wjf2"
server_key: "5e42d03180a79153a737d871f2d3094c"
server_key_iv: "6e036946e0e79a7d580da26c4236cded"
server_key_version: "encryptionkey@0"
I20260923 14:49:35.915280 18093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance:
uuid: "1b1433920de946a492c43ca178f9ad5c"
format_stamp: "Formatted at 2026-09-23 14:49:35 on dist-test-slave-wjf2"
server_key: "5e42d03180a79153a737d871f2d3094c"
server_key_iv: "6e036946e0e79a7d580da26c4236cded"
server_key_version: "encryptionkey@0"
I20260923 14:49:35.919121 18093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 14:49:35.921705 18107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:35.922993 18093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:49:35.923137 18093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/wal
uuid: "1b1433920de946a492c43ca178f9ad5c"
format_stamp: "Formatted at 2026-09-23 14:49:35 on dist-test-slave-wjf2"
server_key: "5e42d03180a79153a737d871f2d3094c"
server_key_iv: "6e036946e0e79a7d580da26c4236cded"
server_key_version: "encryptionkey@0"
I20260923 14:49:35.923244 18093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:35.940361 18093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:35.941015 18093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:35.941207 18093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:35.941784 18093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:35.942691 18093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:35.942770 18093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:35.942862 18093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:35.942911 18093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:35.952013 18093 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.193:43709
I20260923 14:49:35.952080 18221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.193:43709 every 8 connection(s)
I20260923 14:49:35.952901 18093 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/data/info.pb
I20260923 14:49:35.957018 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 18093
I20260923 14:49:35.957154 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-0/wal/instance
I20260923 14:49:35.957499 30731 external_mini_cluster.cc:1428] Setting key 7468fa1baa8dbb798d1df25bd8f92366
I20260923 14:49:35.959754 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.194:0
--local_ip_for_outbound_sockets=127.30.2.194
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42699
--builtin_ntp_servers=127.30.2.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 14:49:35.963106 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.954683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.193:58247 (local address 127.30.2.254:42699)
0923 14:49:35.954856 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:35.954860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:35.956017 (+ 1157us) server_negotiation.cc:407] Connection header received
0923 14:49:35.957443 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:35.957447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:35.957511 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:35.957613 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:35.958826 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.959600 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.961029 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.961285 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.961379 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.962632 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:35.962635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:35.962638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:35.962649 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:35.962668 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:35.962676 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:35.962732 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:35.962923 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 14:49:35.962995 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 14:49:35.964361 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:35.955114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:42699 (local address 127.30.2.193:58247)
0923 14:49:35.955863 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:35.955903 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:35.957200 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:35.957817 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:35.957824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:35.957935 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:35.958596 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.958615 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.959838 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.959843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.960877 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:35.960891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:35.961448 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:35.961453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:35.961671 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:35.962335 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:49:35.962339 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:35.962394 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:49:35.962476 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:35.962767 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:35.962780 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:35.962789 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 14:49:35.962916 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 14:49:35.963213 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":126,"threads_started":1}
I20260923 14:49:35.965760 18222 heartbeater.cc:347] Connected to a master server at 127.30.2.254:42699
I20260923 14:49:35.966044 18222 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:35.966629 18222 heartbeater.cc:511] Master 127.30.2.254:42699 requested a full tablet report, sending...
I20260923 14:49:35.968307 18032 ts_manager.cc:194] Registered new tserver with Master: 1b1433920de946a492c43ca178f9ad5c (127.30.2.193:43709)
I20260923 14:49:35.969769 18032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.193:58247
W20260923 14:49:36.069854 18225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:36.070096 18225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:36.070133 18225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:36.074420 18225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:36.074472 18225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:36.074558 18225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.194
I20260923 14:49:36.079282 18225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.2.194
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.194
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:36.080329 18225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:36.081743 18225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:36.091482 18233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:36.091456 18231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:36.091528 18225 server_base.cc:1034] running on GCE node
W20260923 14:49:36.091456 18230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:36.092101 18225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:36.092743 18225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:36.093973 18225 hybrid_clock.cc:648] HybridClock initialized: now 1790174976093937 us; error 45 us; skew 500 ppm
I20260923 14:49:36.096011 18225 webserver.cc:460] Webserver started at http://127.30.2.194:36819/ using document root <none> and password file <none>
I20260923 14:49:36.096612 18225 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:36.096709 18225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:36.096953 18225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:36.099058 18225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/data/instance:
uuid: "5bf65ecba0fe46019551a30727052805"
format_stamp: "Formatted at 2026-09-23 14:49:36 on dist-test-slave-wjf2"
server_key: "545a621d1e7cd5d4c7be49e4ed500a76"
server_key_iv: "00349ff8de22543ee8d645b6ef9da8d4"
server_key_version: "encryptionkey@0"
I20260923 14:49:36.099677 18225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance:
uuid: "5bf65ecba0fe46019551a30727052805"
format_stamp: "Formatted at 2026-09-23 14:49:36 on dist-test-slave-wjf2"
server_key: "545a621d1e7cd5d4c7be49e4ed500a76"
server_key_iv: "00349ff8de22543ee8d645b6ef9da8d4"
server_key_version: "encryptionkey@0"
I20260923 14:49:36.103693 18225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 14:49:36.106324 18239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:36.107538 18225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:49:36.107676 18225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/wal
uuid: "5bf65ecba0fe46019551a30727052805"
format_stamp: "Formatted at 2026-09-23 14:49:36 on dist-test-slave-wjf2"
server_key: "545a621d1e7cd5d4c7be49e4ed500a76"
server_key_iv: "00349ff8de22543ee8d645b6ef9da8d4"
server_key_version: "encryptionkey@0"
I20260923 14:49:36.107851 18225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:36.138801 18225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:36.139599 18225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:36.139858 18225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:36.140494 18225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:36.141471 18225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:36.141544 18225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:36.141619 18225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:36.141656 18225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:36.150976 18225 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.194:42863
I20260923 14:49:36.151005 18352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.194:42863 every 8 connection(s)
I20260923 14:49:36.151939 18225 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/data/info.pb
I20260923 14:49:36.156656 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 18225
I20260923 14:49:36.156786 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/wal/instance
I20260923 14:49:36.157106 30731 external_mini_cluster.cc:1428] Setting key 7e7048373456fffeed9463cec77a205c
I20260923 14:49:36.159106 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.2.195:0
--local_ip_for_outbound_sockets=127.30.2.195
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42699
--builtin_ntp_servers=127.30.2.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 14:49:36.161943 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.153817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:49627 (local address 127.30.2.254:42699)
0923 14:49:36.153968 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:36.153972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:36.155039 (+ 1067us) server_negotiation.cc:407] Connection header received
0923 14:49:36.156483 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:36.156489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:36.156551 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:36.156657 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:36.157758 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.158376 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.159736 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.160204 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.160329 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.161432 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:36.161436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:36.161439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:36.161451 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:36.161479 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:36.161487 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:36.161564 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:36.161736 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 14:49:36.161800 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:49:36.162806 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.154221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:42699 (local address 127.30.2.194:49627)
0923 14:49:36.154893 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:36.154934 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:36.156270 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:36.156910 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:36.156917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:36.157004 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:36.157610 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.157632 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.158516 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.158520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.159619 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.159629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.160354 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.160357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.160495 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.161177 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:49:36.161179 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:36.161218 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:49:36.161280 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:36.161581 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:36.161597 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:36.161603 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 14:49:36.161722 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 14:49:36.162042 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260923 14:49:36.164011 18353 heartbeater.cc:347] Connected to a master server at 127.30.2.254:42699
I20260923 14:49:36.164263 18353 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:36.164772 18353 heartbeater.cc:511] Master 127.30.2.254:42699 requested a full tablet report, sending...
I20260923 14:49:36.165891 18032 ts_manager.cc:194] Registered new tserver with Master: 5bf65ecba0fe46019551a30727052805 (127.30.2.194:42863)
I20260923 14:49:36.166829 18032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.194:49627
W20260923 14:49:36.271885 18356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:36.272133 18356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:36.272169 18356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:36.276201 18356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:36.276244 18356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:36.276294 18356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.2.195
I20260923 14:49:36.280922 18356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.2.195
--webserver_port=0
--tserver_master_addrs=127.30.2.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.2.195
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:36.281833 18356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:36.283102 18356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:36.293128 18361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:36.293134 18364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:36.293181 18356 server_base.cc:1034] running on GCE node
W20260923 14:49:36.293126 18362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:36.293687 18356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:36.294377 18356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:36.295614 18356 hybrid_clock.cc:648] HybridClock initialized: now 1790174976295595 us; error 35 us; skew 500 ppm
I20260923 14:49:36.297588 18356 webserver.cc:460] Webserver started at http://127.30.2.195:36461/ using document root <none> and password file <none>
I20260923 14:49:36.298190 18356 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:36.298252 18356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:36.298494 18356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:36.300506 18356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/data/instance:
uuid: "44e9259604f942bb8770c4d6a1e6ab64"
format_stamp: "Formatted at 2026-09-23 14:49:36 on dist-test-slave-wjf2"
server_key: "71ebf53120378891fb8b5fa135ed163e"
server_key_iv: "ebf4ffbd364581cd452f0b17c8fe5cb2"
server_key_version: "encryptionkey@0"
I20260923 14:49:36.301061 18356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance:
uuid: "44e9259604f942bb8770c4d6a1e6ab64"
format_stamp: "Formatted at 2026-09-23 14:49:36 on dist-test-slave-wjf2"
server_key: "71ebf53120378891fb8b5fa135ed163e"
server_key_iv: "ebf4ffbd364581cd452f0b17c8fe5cb2"
server_key_version: "encryptionkey@0"
I20260923 14:49:36.304689 18356 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:49:36.307097 18370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:36.308320 18356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:36.308446 18356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/wal
uuid: "44e9259604f942bb8770c4d6a1e6ab64"
format_stamp: "Formatted at 2026-09-23 14:49:36 on dist-test-slave-wjf2"
server_key: "71ebf53120378891fb8b5fa135ed163e"
server_key_iv: "ebf4ffbd364581cd452f0b17c8fe5cb2"
server_key_version: "encryptionkey@0"
I20260923 14:49:36.308553 18356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:36.324556 18356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:36.325202 18356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:36.325392 18356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:36.325945 18356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:49:36.326929 18356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:49:36.326998 18356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:36.327073 18356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:49:36.327126 18356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:36.336329 18356 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.195:45985
I20260923 14:49:36.336449 18484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.195:45985 every 8 connection(s)
I20260923 14:49:36.337224 18356 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/data/info.pb
I20260923 14:49:36.345857 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 18356
I20260923 14:49:36.345983 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/wal/instance
I20260923 14:49:36.346297 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.339058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.195:37991 (local address 127.30.2.254:42699)
0923 14:49:36.339227 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:36.339232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:36.340311 (+ 1079us) server_negotiation.cc:407] Connection header received
0923 14:49:36.341603 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:36.341606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:36.341652 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:36.341726 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:36.342563 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.343091 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.344267 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.344522 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.344632 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.345705 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:36.345708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:36.345710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:36.345723 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:36.345745 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:36.345755 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:36.345831 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:36.346118 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 14:49:36.346180 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 14:49:36.346300 30731 external_mini_cluster.cc:1428] Setting key 5bc1df1b0a1da2bbd1a1758b1fc73c14
I20260923 14:49:36.347282 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.339463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:42699 (local address 127.30.2.195:37991)
0923 14:49:36.340166 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:36.340204 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:36.341406 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:36.341886 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:36.341893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:36.341982 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:49:36.342429 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.342441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.343199 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.343201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.344140 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.344149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.344666 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.344669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.344802 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.345422 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:49:36.345424 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:49:36.345464 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:49:36.345521 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:49:36.345859 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:49:36.345870 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:49:36.345880 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 14:49:36.346033 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 14:49:36.346334 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
I20260923 14:49:36.348397 18485 heartbeater.cc:347] Connected to a master server at 127.30.2.254:42699
I20260923 14:49:36.348599 18485 heartbeater.cc:464] Registering TS with master...
I20260923 14:49:36.349025 18485 heartbeater.cc:511] Master 127.30.2.254:42699 requested a full tablet report, sending...
I20260923 14:49:36.350013 18032 ts_manager.cc:194] Registered new tserver with Master: 44e9259604f942bb8770c4d6a1e6ab64 (127.30.2.195:45985)
I20260923 14:49:36.350569 18032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.2.195:37991
I20260923 14:49:36.361362 30731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:49:36.366966 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.363310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.30.2.254:42699)
0923 14:49:36.363458 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:36.363462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:36.363657 (+ 195us) server_negotiation.cc:407] Connection header received
0923 14:49:36.363708 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:36.363711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:36.363753 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:36.363926 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:36.364165 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.364664 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.365417 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.365640 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.365734 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.366462 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:36.366465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:36.366467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:36.366489 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:36.366507 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:36.366514 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:36.366578 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:36.366692 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 14:49:36.366738 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 14:49:36.369807 18032 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:49:36.372850 18032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:49:36.387535 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.381641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:43709 (local address 127.0.0.1:44968)
0923 14:49:36.382857 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:36.382873 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:36.382968 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:36.383684 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:36.383687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:36.383705 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:36.383848 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.383855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.384971 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.384974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.386162 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.386171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.386982 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.386985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.387087 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.387123 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 14:49:36.387242 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 14:49:36.387376 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260923 14:49:36.387540 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.382415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.30.2.193:43709)
0923 14:49:36.383232 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:36.383248 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:36.383267 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:49:36.383331 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:36.383335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:36.383525 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:36.383995 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:36.384129 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.384847 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.386286 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.386853 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.386977 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.387170 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:36.387280 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 14:49:36.387376 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":98,"threads_started":1}
I20260923 14:49:36.389024 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.381979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:45985 (local address 127.0.0.1:58020)
0923 14:49:36.383033 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:36.383047 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:36.383130 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:36.383846 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:36.383849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:36.383864 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:36.383950 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.383956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.385502 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.385505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.386923 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.386936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.388461 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.388465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.388600 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.388633 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 14:49:36.388735 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 14:49:36.388841 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":31,"thread_start_us":101,"threads_started":1}
I20260923 14:49:36.389439 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.381697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.30.2.194:42863)
0923 14:49:36.382986 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:36.383007 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:36.383036 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:49:36.383125 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:36.383132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:36.383350 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:36.383543 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:36.383903 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.385277 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.387102 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.388283 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.388424 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.388612 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:36.389106 (+ 494us) server_negotiation.cc:299] Negotiation successful
0923 14:49:36.389251 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1154,"thread_start_us":169,"threads_started":1}
I20260923 14:49:36.389022 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.381505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.194:42863 (local address 127.0.0.1:58090)
0923 14:49:36.382301 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:36.382359 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:36.382505 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:36.383537 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:36.383544 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:36.383571 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:36.383745 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.383753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.385449 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.385453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.386952 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.386960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.388447 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.388452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.388716 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.388744 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 14:49:36.388821 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 14:49:36.388881 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":315,"thread_start_us":76,"threads_started":1}
I20260923 14:49:36.389596 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.382427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.30.2.195:45985)
0923 14:49:36.382872 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:36.382886 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:36.383222 (+ 336us) server_negotiation.cc:407] Connection header received
0923 14:49:36.383321 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:36.383326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:36.383525 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:36.383676 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:36.384070 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.385365 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.387102 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.388037 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.388176 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.388385 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:36.389285 (+ 900us) server_negotiation.cc:299] Negotiation successful
0923 14:49:36.389392 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":132,"threads_started":1}
I20260923 14:49:36.390827 18155 tablet_service.cc:1467] Processing CreateTablet for tablet 72d6d63d09084a238fee7426c1250b20 (DEFAULT_TABLE table=test-table [id=034caf2971f04ef698a99096c792ea7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:36.392066 18155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d6d63d09084a238fee7426c1250b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:36.392450 18419 tablet_service.cc:1467] Processing CreateTablet for tablet 72d6d63d09084a238fee7426c1250b20 (DEFAULT_TABLE table=test-table [id=034caf2971f04ef698a99096c792ea7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:36.392501 18287 tablet_service.cc:1467] Processing CreateTablet for tablet 72d6d63d09084a238fee7426c1250b20 (DEFAULT_TABLE table=test-table [id=034caf2971f04ef698a99096c792ea7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:49:36.393365 18419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d6d63d09084a238fee7426c1250b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:36.393567 18287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d6d63d09084a238fee7426c1250b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:36.399531 18503 tablet_bootstrap.cc:492] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805: Bootstrap starting.
I20260923 14:49:36.399972 18504 tablet_bootstrap.cc:492] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c: Bootstrap starting.
I20260923 14:49:36.400555 18506 tablet_bootstrap.cc:492] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: Bootstrap starting.
I20260923 14:49:36.402266 18504 tablet_bootstrap.cc:654] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:36.402256 18506 tablet_bootstrap.cc:654] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:36.403079 18506 log.cc:824] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:36.403124 18504 log.cc:824] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:36.404268 18503 tablet_bootstrap.cc:654] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:36.405027 18504 tablet_bootstrap.cc:492] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c: No bootstrap required, opened a new log
I20260923 14:49:36.405041 18503 log.cc:824] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:36.405159 18506 tablet_bootstrap.cc:492] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: No bootstrap required, opened a new log
I20260923 14:49:36.405259 18504 ts_tablet_manager.cc:1397] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:49:36.405354 18506 ts_tablet_manager.cc:1397] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:49:36.407419 18503 tablet_bootstrap.cc:492] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805: No bootstrap required, opened a new log
I20260923 14:49:36.407604 18503 ts_tablet_manager.cc:1397] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260923 14:49:36.408653 18506 raft_consensus.cc:348] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf65ecba0fe46019551a30727052805" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42863 } } peers { permanent_uuid: "1b1433920de946a492c43ca178f9ad5c" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43709 } } peers { permanent_uuid: "44e9259604f942bb8770c4d6a1e6ab64" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45985 } }
I20260923 14:49:36.408879 18506 raft_consensus.cc:374] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:36.408934 18506 raft_consensus.cc:729] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e9259604f942bb8770c4d6a1e6ab64, State: Initialized, Role: FOLLOWER
I20260923 14:49:36.409369 18506 consensus_queue.cc:260] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf65ecba0fe46019551a30727052805" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42863 } } peers { permanent_uuid: "1b1433920de946a492c43ca178f9ad5c" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43709 } } peers { permanent_uuid: "44e9259604f942bb8770c4d6a1e6ab64" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45985 } }
I20260923 14:49:36.410017 18485 heartbeater.cc:503] Master 127.30.2.254:42699 was elected leader, sending a full tablet report...
I20260923 14:49:36.410404 18506 ts_tablet_manager.cc:1428] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:49:36.409601 18504 raft_consensus.cc:348] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf65ecba0fe46019551a30727052805" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42863 } } peers { permanent_uuid: "1b1433920de946a492c43ca178f9ad5c" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43709 } } peers { permanent_uuid: "44e9259604f942bb8770c4d6a1e6ab64" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45985 } }
I20260923 14:49:36.410841 18504 raft_consensus.cc:374] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:36.410914 18504 raft_consensus.cc:729] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1433920de946a492c43ca178f9ad5c, State: Initialized, Role: FOLLOWER
I20260923 14:49:36.411403 18504 consensus_queue.cc:260] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf65ecba0fe46019551a30727052805" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42863 } } peers { permanent_uuid: "1b1433920de946a492c43ca178f9ad5c" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43709 } } peers { permanent_uuid: "44e9259604f942bb8770c4d6a1e6ab64" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45985 } }
I20260923 14:49:36.411561 18503 raft_consensus.cc:348] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf65ecba0fe46019551a30727052805" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42863 } } peers { permanent_uuid: "1b1433920de946a492c43ca178f9ad5c" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43709 } } peers { permanent_uuid: "44e9259604f942bb8770c4d6a1e6ab64" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45985 } }
I20260923 14:49:36.411819 18503 raft_consensus.cc:374] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:36.411880 18503 raft_consensus.cc:729] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf65ecba0fe46019551a30727052805, State: Initialized, Role: FOLLOWER
I20260923 14:49:36.412436 18504 ts_tablet_manager.cc:1428] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260923 14:49:36.412420 18503 consensus_queue.cc:260] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf65ecba0fe46019551a30727052805" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42863 } } peers { permanent_uuid: "1b1433920de946a492c43ca178f9ad5c" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43709 } } peers { permanent_uuid: "44e9259604f942bb8770c4d6a1e6ab64" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45985 } }
I20260923 14:49:36.412784 18222 heartbeater.cc:503] Master 127.30.2.254:42699 was elected leader, sending a full tablet report...
I20260923 14:49:36.413292 18353 heartbeater.cc:503] Master 127.30.2.254:42699 was elected leader, sending a full tablet report...
I20260923 14:49:36.413462 18503 ts_tablet_manager.cc:1428] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
W20260923 14:49:36.454144 18223 tablet.cc:2369] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:49:36.565059 18512 raft_consensus.cc:482] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:49:36.565376 18512 raft_consensus.cc:504] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf65ecba0fe46019551a30727052805" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42863 } } peers { permanent_uuid: "1b1433920de946a492c43ca178f9ad5c" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43709 } } peers { permanent_uuid: "44e9259604f942bb8770c4d6a1e6ab64" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45985 } }
I20260923 14:49:36.567144 18512 leader_election.cc:288] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b1433920de946a492c43ca178f9ad5c (127.30.2.193:43709), 44e9259604f942bb8770c4d6a1e6ab64 (127.30.2.195:45985)
I20260923 14:49:36.571362 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.567605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:54073 (local address 127.30.2.193:43709)
0923 14:49:36.567869 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:36.567875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:36.567891 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:49:36.567957 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:36.567962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:36.568018 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:36.568092 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:36.568474 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.569013 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.570347 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.570718 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.570818 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.570916 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:36.571145 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 14:49:36.571217 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260923 14:49:36.571398 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.567394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.195:45985 (local address 127.30.2.194:45527)
0923 14:49:36.567896 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:36.567910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:36.568007 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:36.568305 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:36.568307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:36.568323 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:36.568455 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.568462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.569294 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.569297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.570097 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.570105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.570925 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.570929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.571034 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.571051 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:49:36.571145 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 14:49:36.571248 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":29,"thread_start_us":119,"threads_started":1}
I20260923 14:49:36.571413 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.567370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:43709 (local address 127.30.2.194:54073)
0923 14:49:36.567573 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:36.567593 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 14:49:36.567699 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:49:36.568125 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:49:36.568128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:49:36.568149 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:49:36.568333 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.568343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.569139 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.569143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.570124 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:49:36.570146 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.570856 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.570863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:49:36.570982 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.571019 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 14:49:36.571107 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 14:49:36.571249 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":27}
I20260923 14:49:36.571413 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.567544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.2.194:45527 (local address 127.30.2.195:45985)
0923 14:49:36.567839 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:36.567847 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:36.568010 (+ 163us) server_negotiation.cc:407] Connection header received
0923 14:49:36.568106 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:36.568110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:36.568187 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:36.568285 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:49:36.568580 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.569170 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.570239 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.570755 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.570921 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.570993 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:36.571187 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 14:49:36.571266 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260923 14:49:36.572098 18439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d6d63d09084a238fee7426c1250b20" candidate_uuid: "5bf65ecba0fe46019551a30727052805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e9259604f942bb8770c4d6a1e6ab64" is_pre_election: true
I20260923 14:49:36.572203 18176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d6d63d09084a238fee7426c1250b20" candidate_uuid: "5bf65ecba0fe46019551a30727052805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1433920de946a492c43ca178f9ad5c" is_pre_election: true
I20260923 14:49:36.572448 18439 raft_consensus.cc:2454] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf65ecba0fe46019551a30727052805 in term 0.
I20260923 14:49:36.572585 18176 raft_consensus.cc:2454] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf65ecba0fe46019551a30727052805 in term 0.
I20260923 14:49:36.572943 18243 leader_election.cc:302] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44e9259604f942bb8770c4d6a1e6ab64, 5bf65ecba0fe46019551a30727052805; no voters:
I20260923 14:49:36.573282 18512 raft_consensus.cc:2781] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:49:36.573372 18512 raft_consensus.cc:482] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:49:36.573432 18512 raft_consensus.cc:3037] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:36.574720 18512 raft_consensus.cc:504] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf65ecba0fe46019551a30727052805" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42863 } } peers { permanent_uuid: "1b1433920de946a492c43ca178f9ad5c" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43709 } } peers { permanent_uuid: "44e9259604f942bb8770c4d6a1e6ab64" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45985 } }
I20260923 14:49:36.575122 18512 leader_election.cc:288] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [CANDIDATE]: Term 1 election: Requested vote from peers 1b1433920de946a492c43ca178f9ad5c (127.30.2.193:43709), 44e9259604f942bb8770c4d6a1e6ab64 (127.30.2.195:45985)
I20260923 14:49:36.575511 18176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d6d63d09084a238fee7426c1250b20" candidate_uuid: "5bf65ecba0fe46019551a30727052805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1433920de946a492c43ca178f9ad5c"
I20260923 14:49:36.575521 18439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d6d63d09084a238fee7426c1250b20" candidate_uuid: "5bf65ecba0fe46019551a30727052805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e9259604f942bb8770c4d6a1e6ab64"
I20260923 14:49:36.575614 18176 raft_consensus.cc:3037] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:36.575668 18439 raft_consensus.cc:3037] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:36.576725 18176 raft_consensus.cc:2454] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf65ecba0fe46019551a30727052805 in term 1.
I20260923 14:49:36.576951 18439 raft_consensus.cc:2454] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf65ecba0fe46019551a30727052805 in term 1.
I20260923 14:49:36.577082 18240 leader_election.cc:302] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1433920de946a492c43ca178f9ad5c, 5bf65ecba0fe46019551a30727052805; no voters:
I20260923 14:49:36.577277 18512 raft_consensus.cc:2781] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:36.577466 18512 raft_consensus.cc:686] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 1 LEADER]: Becoming Leader. State: Replica: 5bf65ecba0fe46019551a30727052805, State: Running, Role: LEADER
I20260923 14:49:36.577879 18512 consensus_queue.cc:237] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf65ecba0fe46019551a30727052805" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42863 } } peers { permanent_uuid: "1b1433920de946a492c43ca178f9ad5c" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43709 } } peers { permanent_uuid: "44e9259604f942bb8770c4d6a1e6ab64" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45985 } }
I20260923 14:49:36.580981 18031 catalog_manager.cc:5495] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf65ecba0fe46019551a30727052805 (127.30.2.194). New cstate: current_term: 1 leader_uuid: "5bf65ecba0fe46019551a30727052805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf65ecba0fe46019551a30727052805" member_type: VOTER last_known_addr { host: "127.30.2.194" port: 42863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b1433920de946a492c43ca178f9ad5c" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e9259604f942bb8770c4d6a1e6ab64" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45985 } health_report { overall_health: UNKNOWN } } }
W20260923 14:49:36.588349 18486 tablet.cc:2369] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:49:36.622944 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.618737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.30.2.194:42863)
0923 14:49:36.618930 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:36.618933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:36.618947 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:49:36.619025 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:36.619028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:36.619085 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:36.619175 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:49:36.619556 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.620182 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.621229 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:36.621478 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:36.621599 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:36.621967 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:49:36.622510 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:49:36.622655 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:36.622710 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 14:49:36.622766 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260923 14:49:36.629212 18439 raft_consensus.cc:1261] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64 [term 1 FOLLOWER]: Refusing update from remote peer 5bf65ecba0fe46019551a30727052805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:36.629212 18176 raft_consensus.cc:1261] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c [term 1 FOLLOWER]: Refusing update from remote peer 5bf65ecba0fe46019551a30727052805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:49:36.629930 18512 consensus_queue.cc:1035] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1433920de946a492c43ca178f9ad5c" member_type: VOTER last_known_addr { host: "127.30.2.193" port: 43709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:36.630141 18515 consensus_queue.cc:1035] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e9259604f942bb8770c4d6a1e6ab64" member_type: VOTER last_known_addr { host: "127.30.2.195" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:49:36.648399 18032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:36.648578 18032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:49:36.651181 18032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94d7570387d648f59e0e2b1660ba907a: Sending DeleteTablet for 3 replicas of tablet 72d6d63d09084a238fee7426c1250b20
I20260923 14:49:36.652050 18287 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d6d63d09084a238fee7426c1250b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:36 UTC) from {username='slave'} at 127.0.0.1:58090
I20260923 14:49:36.652055 18155 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d6d63d09084a238fee7426c1250b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:36 UTC) from {username='slave'} at 127.0.0.1:44968
I20260923 14:49:36.652557 18419 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d6d63d09084a238fee7426c1250b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:49:36 UTC) from {username='slave'} at 127.0.0.1:58020
I20260923 14:49:36.652695 18528 tablet_replica.cc:330] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c: stopping tablet replica
I20260923 14:49:36.653035 18530 tablet_replica.cc:330] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: stopping tablet replica
I20260923 14:49:36.653307 18530 raft_consensus.cc:2229] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:49:36.653383 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 18093
I20260923 14:49:36.653621 18530 raft_consensus.cc:2258] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:36.653718 18528 raft_consensus.cc:2229] T 72d6d63d09084a238fee7426c1250b20 P 1b1433920de946a492c43ca178f9ad5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:49:36.654047 18529 tablet_replica.cc:330] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805: stopping tablet replica
I20260923 14:49:36.654318 18529 raft_consensus.cc:2229] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:49:36.654685 18529 raft_consensus.cc:2258] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:49:36.655282 18530 ts_tablet_manager.cc:1905] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:49:36.655989 18529 ts_tablet_manager.cc:1905] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:49:36.658099 18530 ts_tablet_manager.cc:1918] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:49:36.658221 18530 log.cc:1196] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-2/wal/wals/72d6d63d09084a238fee7426c1250b20
I20260923 14:49:36.658520 18530 ts_tablet_manager.cc:1939] T 72d6d63d09084a238fee7426c1250b20 P 44e9259604f942bb8770c4d6a1e6ab64: Deleting consensus metadata
I20260923 14:49:36.658767 18529 ts_tablet_manager.cc:1918] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:49:36.658843 18529 log.cc:1196] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805: Deleting WAL directory at /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174723338690-30731-0/minicluster-data/ts-1/wal/wals/72d6d63d09084a238fee7426c1250b20
I20260923 14:49:36.659126 18529 ts_tablet_manager.cc:1939] T 72d6d63d09084a238fee7426c1250b20 P 5bf65ecba0fe46019551a30727052805: Deleting consensus metadata
I20260923 14:49:36.660049 18019 catalog_manager.cc:4841] TS 44e9259604f942bb8770c4d6a1e6ab64 (127.30.2.195:45985): tablet 72d6d63d09084a238fee7426c1250b20 (table test-table [id=034caf2971f04ef698a99096c792ea7f]) successfully deleted
I20260923 14:49:36.660236 18017 catalog_manager.cc:4841] TS 5bf65ecba0fe46019551a30727052805 (127.30.2.194:42863): tablet 72d6d63d09084a238fee7426c1250b20 (table test-table [id=034caf2971f04ef698a99096c792ea7f]) successfully deleted
W20260923 14:49:36.664640 18016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.2.193:43709 (error 108)
I20260923 14:49:36.664840 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 18225
I20260923 14:49:36.666288 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:36.665640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.193:43709 (local address 127.0.0.1:44984)
0923 14:49:36.666132 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 14:49:36.666205 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.193:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366}
W20260923 14:49:36.666676 18016 catalog_manager.cc:4584] TS 1b1433920de946a492c43ca178f9ad5c (127.30.2.193:43709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72d6d63d09084a238fee7426c1250b20: Network error: Client connection negotiation failed: client connection to 127.30.2.193:43709: connect: Connection refused (error 111)
I20260923 14:49:36.671187 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 18356
I20260923 14:49:36.676975 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 18001
2026-09-23T14:49:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1193 ms)
[----------] 6 tests from EncryptionPolicyTest (10754 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:36 dist-test-slave-wjf2 krb5kdc[18534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:36 dist-test-slave-wjf2 krb5kdc[18534](info): set up 2 sockets
Sep 23 14:49:36 dist-test-slave-wjf2 krb5kdc[18534](info): commencing operation
krb5kdc: starting...
W20260923 14:49:37.713326 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:37 dist-test-slave-wjf2 krb5kdc[18534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:37.839118 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:37.948720 18547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:37.949013 18547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:37.949085 18547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:37.953059 18547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:37.953131 18547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:37.953162 18547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:37.953188 18547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:37.953203 18547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:37.962083 18547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:37.963538 18547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:37.965041 18547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:37.976728 18553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:37.977048 18547 server_base.cc:1034] running on GCE node
W20260923 14:49:37.977167 18555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:37.978876 18552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:37.979732 18547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:37.981354 18547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:37.982563 18547 hybrid_clock.cc:648] HybridClock initialized: now 1790174977982535 us; error 40 us; skew 500 ppm
Sep 23 14:49:37 dist-test-slave-wjf2 krb5kdc[18534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:37.986196 18547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:37.987495 18547 webserver.cc:460] Webserver started at http://127.0.0.1:37679/ using document root <none> and password file <none>
I20260923 14:49:37.988332 18547 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:37.988422 18547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:37.988646 18547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:37.991554 18547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "dcbbc09943db4c8784bdf721bb828864"
format_stamp: "Formatted at 2026-09-23 14:49:37 on dist-test-slave-wjf2"
server_key: "4c5f8cba09873b3a58ff1d6eee13aef7"
server_key_iv: "8854416f3f1d89690fdd0cefb282ed21"
server_key_version: "encryptionkey@0"
I20260923 14:49:37.992300 18547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "dcbbc09943db4c8784bdf721bb828864"
format_stamp: "Formatted at 2026-09-23 14:49:37 on dist-test-slave-wjf2"
server_key: "4c5f8cba09873b3a58ff1d6eee13aef7"
server_key_iv: "8854416f3f1d89690fdd0cefb282ed21"
server_key_version: "encryptionkey@0"
I20260923 14:49:37.996485 18547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 14:49:37.999774 18562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:38.001487 18547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:49:38.001631 18547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "dcbbc09943db4c8784bdf721bb828864"
format_stamp: "Formatted at 2026-09-23 14:49:37 on dist-test-slave-wjf2"
server_key: "4c5f8cba09873b3a58ff1d6eee13aef7"
server_key_iv: "8854416f3f1d89690fdd0cefb282ed21"
server_key_version: "encryptionkey@0"
I20260923 14:49:38.001750 18547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:38.015450 18547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:38.018271 18547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:38.018498 18547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:38.026005 18547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41023
I20260923 14:49:38.026033 18614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41023 every 8 connection(s)
I20260923 14:49:38.027009 18547 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:38.029760 18615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:38.035175 18615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864: Bootstrap starting.
I20260923 14:49:38.035437 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 18547
I20260923 14:49:38.035568 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:38.035898 30731 external_mini_cluster.cc:1428] Setting key 6675a69023ad111072d53744c43984dd
I20260923 14:49:38.037825 18615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:38.038540 18615 log.cc:824] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:38.040652 18615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864: No bootstrap required, opened a new log
Sep 23 14:49:38 dist-test-slave-wjf2 krb5kdc[18534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 14:49:38.045147 18615 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbbc09943db4c8784bdf721bb828864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260923 14:49:38.045387 18615 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:38.045463 18615 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbbc09943db4c8784bdf721bb828864, State: Initialized, Role: FOLLOWER
I20260923 14:49:38.045933 18615 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbbc09943db4c8784bdf721bb828864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260923 14:49:38.046108 18615 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:38.046176 18615 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:38.046283 18615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:38.047447 18615 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbbc09943db4c8784bdf721bb828864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260923 14:49:38.047847 18615 leader_election.cc:302] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcbbc09943db4c8784bdf721bb828864; no voters:
I20260923 14:49:38.048197 18615 leader_election.cc:288] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:38.048262 18620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:38.048467 18620 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [term 1 LEADER]: Becoming Leader. State: Replica: dcbbc09943db4c8784bdf721bb828864, State: Running, Role: LEADER
I20260923 14:49:38.048759 18620 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbbc09943db4c8784bdf721bb828864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260923 14:49:38.049489 18615 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:38.049929 18622 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcbbc09943db4c8784bdf721bb828864. Latest consensus state: current_term: 1 leader_uuid: "dcbbc09943db4c8784bdf721bb828864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbbc09943db4c8784bdf721bb828864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } } }
I20260923 14:49:38.050052 18622 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:38.050388 18621 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcbbc09943db4c8784bdf721bb828864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbbc09943db4c8784bdf721bb828864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } } }
I20260923 14:49:38.050519 18621 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:38.051632 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:38.037261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59377 (local address 127.0.0.1:41023)
0923 14:49:38.037671 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:38.037685 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:38.037725 (+ 40us) server_negotiation.cc:407] Connection header received
0923 14:49:38.038389 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:38.038408 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:38.038778 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:38.039134 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:38.039998 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:38.041045 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:38.042211 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:38.042665 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:38.042775 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:38.045723 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:38.045728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:38.045744 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:38.045776 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:38.047881 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:38.048626 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:38.048632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:38.048638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:38.048774 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:38.049172 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:38.049175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:38.049176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:38.049602 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:38.049845 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:38.050433 (+ 588us) server_negotiation.cc:299] Negotiation successful
0923 14:49:38.050668 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":122,"threads_started":1}
I20260923 14:49:38.055248 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 14:49:38.055236 18635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:49:38.055328 18635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:49:38.055449 18624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:38.056406 18624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:38.062326 18624 catalog_manager.cc:1347] Generated new cluster ID: fcd3d2b608614e468e98d0736cbd9cbe
I20260923 14:49:38.062388 18624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:38.066090 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:38.057344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58117 (local address 127.0.0.1:41023)
0923 14:49:38.057457 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:38.057461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:38.057824 (+ 363us) server_negotiation.cc:407] Connection header received
0923 14:49:38.057968 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:38.057971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:38.058011 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:38.058100 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:38.058910 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:38.059834 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:38.060766 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:38.061140 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:38.061252 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:38.062839 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:38.062843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:38.062846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:38.062870 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:38.064365 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:38.064833 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:38.064837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:38.064839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:38.064918 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:38.065239 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:38.065243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:38.065244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:38.065412 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:38.065564 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:38.065837 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 14:49:38.065925 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:49:38.086212 18624 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:38.087236 18624 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:38.097612 18624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcbbc09943db4c8784bdf721bb828864: Generated new TSK 0
I20260923 14:49:38.098501 18624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:38.109028 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 18547
W20260923 14:49:38.117054 18532 connection.cc:535] client connection to 127.0.0.1:41023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41023): Connection reset by peer (error 104)
2026-09-23T14:49:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:38 dist-test-slave-wjf2 krb5kdc[18646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:38 dist-test-slave-wjf2 krb5kdc[18646](info): set up 2 sockets
Sep 23 14:49:38 dist-test-slave-wjf2 krb5kdc[18646](info): commencing operation
krb5kdc: starting...
W20260923 14:49:39.145256 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:39 dist-test-slave-wjf2 krb5kdc[18646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:39.270213 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:39.382009 18659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:39.382266 18659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:39.382308 18659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:39.386325 18659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:39.386380 18659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:39.386400 18659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:39.386416 18659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:39.386432 18659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:39.391634 18659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46579
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:39.392701 18659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:39.393668 18659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:39.402702 18667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:39.402828 18665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:39.402997 18659 server_base.cc:1034] running on GCE node
W20260923 14:49:39.402706 18664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:39.403688 18659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:39.404884 18659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:39.406097 18659 hybrid_clock.cc:648] HybridClock initialized: now 1790174979406072 us; error 38 us; skew 500 ppm
Sep 23 14:49:39 dist-test-slave-wjf2 krb5kdc[18646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:39.409066 18659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:39.410048 18659 webserver.cc:460] Webserver started at http://127.0.0.1:39757/ using document root <none> and password file <none>
I20260923 14:49:39.410656 18659 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:39.410758 18659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:39.411013 18659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:39.413259 18659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "d76f3178118d4f08999ca04472324b5e"
format_stamp: "Formatted at 2026-09-23 14:49:39 on dist-test-slave-wjf2"
server_key: "0fdf9847754017f919a9a1268397df43"
server_key_iv: "fb5150a48e44a77038be6f194cbb6316"
server_key_version: "encryptionkey@0"
I20260923 14:49:39.413868 18659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "d76f3178118d4f08999ca04472324b5e"
format_stamp: "Formatted at 2026-09-23 14:49:39 on dist-test-slave-wjf2"
server_key: "0fdf9847754017f919a9a1268397df43"
server_key_iv: "fb5150a48e44a77038be6f194cbb6316"
server_key_version: "encryptionkey@0"
I20260923 14:49:39.417940 18659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:49:39.420639 18674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:39.422075 18659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:49:39.422344 18659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "d76f3178118d4f08999ca04472324b5e"
format_stamp: "Formatted at 2026-09-23 14:49:39 on dist-test-slave-wjf2"
server_key: "0fdf9847754017f919a9a1268397df43"
server_key_iv: "fb5150a48e44a77038be6f194cbb6316"
server_key_version: "encryptionkey@0"
I20260923 14:49:39.422468 18659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:39.461503 18659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:39.464637 18659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:39.464870 18659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:39.472766 18659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46579
I20260923 14:49:39.472786 18726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46579 every 8 connection(s)
I20260923 14:49:39.473793 18659 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:39.476337 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 18659
I20260923 14:49:39.476480 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:39.476751 18727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:39.476806 30731 external_mini_cluster.cc:1428] Setting key 25f5b26d5f6a3dd333838b0ca9bdf569
I20260923 14:49:39.483076 18727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e: Bootstrap starting.
I20260923 14:49:39.485486 18727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:39.486194 18727 log.cc:824] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:39.486539 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:39.478266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51883 (local address 127.0.0.1:46579)
0923 14:49:39.478742 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:39.478767 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:39.478805 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:49:39.479524 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:39.479542 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:39.480018 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:39.480370 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:39.480827 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:39.481949 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:39.482924 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:39.483501 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:39.483693 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:39.484758 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:39.484763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:39.484777 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:39.485155 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:39.485207 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:39.485230 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:39.485380 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:39.485570 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 14:49:39.485717 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":327,"thread_start_us":153,"threads_started":1}
I20260923 14:49:39.488240 18727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e: No bootstrap required, opened a new log
I20260923 14:49:39.489682 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:39.492877 18727 raft_consensus.cc:348] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76f3178118d4f08999ca04472324b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } }
I20260923 14:49:39.493153 18727 raft_consensus.cc:374] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:39.493221 18727 raft_consensus.cc:729] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76f3178118d4f08999ca04472324b5e, State: Initialized, Role: FOLLOWER
I20260923 14:49:39.493686 18727 consensus_queue.cc:260] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76f3178118d4f08999ca04472324b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } }
I20260923 14:49:39.493858 18727 raft_consensus.cc:388] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:39.493927 18727 raft_consensus.cc:482] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:39.494025 18727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:39.495062 18727 raft_consensus.cc:504] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76f3178118d4f08999ca04472324b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } }
I20260923 14:49:39.495472 18727 leader_election.cc:302] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d76f3178118d4f08999ca04472324b5e; no voters:
I20260923 14:49:39.495828 18727 leader_election.cc:288] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:39.495930 18737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:39.496248 18737 raft_consensus.cc:686] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [term 1 LEADER]: Becoming Leader. State: Replica: d76f3178118d4f08999ca04472324b5e, State: Running, Role: LEADER
I20260923 14:49:39.496655 18737 consensus_queue.cc:237] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76f3178118d4f08999ca04472324b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } }
I20260923 14:49:39.497100 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:39.492042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53287 (local address 127.0.0.1:46579)
0923 14:49:39.492188 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:39.492192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:39.492419 (+ 227us) server_negotiation.cc:407] Connection header received
0923 14:49:39.492513 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:39.492516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:39.492556 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:39.492626 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:39.492911 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:39.493558 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:39.494765 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:39.495018 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:39.495131 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:39.496268 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:39.496272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:39.496276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:39.496291 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:39.496322 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:39.496337 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:39.496434 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:39.496930 (+ 496us) server_negotiation.cc:299] Negotiation successful
0923 14:49:39.496970 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:49:39.497221 18727 sys_catalog.cc:564] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:39.498842 18739 sys_catalog.cc:455] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d76f3178118d4f08999ca04472324b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76f3178118d4f08999ca04472324b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } } }
I20260923 14:49:39.498978 18739 sys_catalog.cc:458] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:39.499195 18737 sys_catalog.cc:455] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d76f3178118d4f08999ca04472324b5e. Latest consensus state: current_term: 1 leader_uuid: "d76f3178118d4f08999ca04472324b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76f3178118d4f08999ca04472324b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } } }
I20260923 14:49:39.500111 18737 sys_catalog.cc:458] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e [sys.catalog]: This master's current role is: LEADER
W20260923 14:49:39.501713 18752 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:49:39.501789 18752 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:49:39.501875 18751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:39.502696 18751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:39.508371 18751 catalog_manager.cc:1347] Generated new cluster ID: 1fded7570fc347abbeae21cb7aeb3912
I20260923 14:49:39.508436 18751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:39.516037 18751 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:39.517264 18751 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:39.534994 18751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d76f3178118d4f08999ca04472324b5e: Generated new TSK 0
I20260923 14:49:39.535929 18751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:39.541069 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 18659
W20260923 14:49:39.547197 18644 connection.cc:535] client connection to 127.0.0.1:46579 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46579): Connection reset by peer (error 104)
2026-09-23T14:49:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:39 dist-test-slave-wjf2 krb5kdc[18758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:39 dist-test-slave-wjf2 krb5kdc[18758](info): set up 2 sockets
Sep 23 14:49:39 dist-test-slave-wjf2 krb5kdc[18758](info): commencing operation
krb5kdc: starting...
W20260923 14:49:40.577206 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:40 dist-test-slave-wjf2 krb5kdc[18758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:40.698451 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:40.807230 18772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:40.807518 18772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:40.807596 18772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:40.811585 18772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:40.811678 18772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:40.811717 18772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:40.811775 18772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:40.811836 18772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:40.817277 18772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:40.818331 18772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:40.819367 18772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:40.828047 18778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:40.828088 18777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:40.828227 18772 server_base.cc:1034] running on GCE node
W20260923 14:49:40.828078 18780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:40.828940 18772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:40.829919 18772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:40.831121 18772 hybrid_clock.cc:648] HybridClock initialized: now 1790174980831102 us; error 26 us; skew 500 ppm
Sep 23 14:49:40 dist-test-slave-wjf2 krb5kdc[18758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:40.833779 18772 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:40.834597 18772 webserver.cc:460] Webserver started at http://127.0.0.1:33769/ using document root <none> and password file <none>
I20260923 14:49:40.835167 18772 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:40.835225 18772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:40.835498 18772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:40.837564 18772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "1f9e7c8e2ef54085a3f74fad2b413f79"
format_stamp: "Formatted at 2026-09-23 14:49:40 on dist-test-slave-wjf2"
server_key: "8b93cdea19fa272baba8c8506a212e03"
server_key_iv: "dc0c5befcb36e114b18e6a424263cb52"
server_key_version: "encryptionkey@0"
I20260923 14:49:40.838119 18772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "1f9e7c8e2ef54085a3f74fad2b413f79"
format_stamp: "Formatted at 2026-09-23 14:49:40 on dist-test-slave-wjf2"
server_key: "8b93cdea19fa272baba8c8506a212e03"
server_key_iv: "dc0c5befcb36e114b18e6a424263cb52"
server_key_version: "encryptionkey@0"
I20260923 14:49:40.841748 18772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:49:40.844261 18787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:40.845541 18772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:40.845690 18772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "1f9e7c8e2ef54085a3f74fad2b413f79"
format_stamp: "Formatted at 2026-09-23 14:49:40 on dist-test-slave-wjf2"
server_key: "8b93cdea19fa272baba8c8506a212e03"
server_key_iv: "dc0c5befcb36e114b18e6a424263cb52"
server_key_version: "encryptionkey@0"
I20260923 14:49:40.845798 18772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:40.850818 18772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:40.853904 18772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:40.854085 18772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:40.861147 18772 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37665
I20260923 14:49:40.861176 18839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37665 every 8 connection(s)
I20260923 14:49:40.862071 18772 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:40.864140 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 18772
I20260923 14:49:40.864288 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:40.864635 30731 external_mini_cluster.cc:1428] Setting key a1b9e7c033d00d018182e27a400b0429
I20260923 14:49:40.865286 18840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:40.869659 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:40.865980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34429 (local address 127.0.0.1:37665)
0923 14:49:40.866428 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:40.866446 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:40.866495 (+ 49us) server_negotiation.cc:407] Connection header received
0923 14:49:40.867113 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:40.867128 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:40.867387 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:40.867711 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:40.868525 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:40.868532 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:40.868555 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:40.868566 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:40.868619 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:40.868638 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:40.868781 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:40.868865 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 14:49:40.869030 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":163,"threads_started":1}
I20260923 14:49:40.871999 18840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79: Bootstrap starting.
I20260923 14:49:40.872609 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:40.874502 18840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:40.875344 18840 log.cc:824] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:40.876362 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:40.874596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49283 (local address 127.0.0.1:37665)
0923 14:49:40.874714 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:40.874718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:40.874901 (+ 183us) server_negotiation.cc:407] Connection header received
0923 14:49:40.875093 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:40.875097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:40.875138 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:40.875219 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:40.875972 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:40.875976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:40.875978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:40.875990 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:40.876017 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:40.876035 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:40.876123 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:40.876209 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 14:49:40.876248 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:49:40.877682 18840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79: No bootstrap required, opened a new log
I20260923 14:49:40.881309 18840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e7c8e2ef54085a3f74fad2b413f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260923 14:49:40.881531 18840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:40.881633 18840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9e7c8e2ef54085a3f74fad2b413f79, State: Initialized, Role: FOLLOWER
I20260923 14:49:40.882190 18840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e7c8e2ef54085a3f74fad2b413f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260923 14:49:40.882403 18840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:40.882483 18840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:40.882586 18840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:40.883530 18840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e7c8e2ef54085a3f74fad2b413f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260923 14:49:40.883996 18840 leader_election.cc:302] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f9e7c8e2ef54085a3f74fad2b413f79; no voters:
I20260923 14:49:40.884325 18840 leader_election.cc:288] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:40.884444 18850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:40.884696 18850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9e7c8e2ef54085a3f74fad2b413f79, State: Running, Role: LEADER
I20260923 14:49:40.885010 18850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e7c8e2ef54085a3f74fad2b413f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260923 14:49:40.885576 18840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:40.886751 18852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f9e7c8e2ef54085a3f74fad2b413f79. Latest consensus state: current_term: 1 leader_uuid: "1f9e7c8e2ef54085a3f74fad2b413f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e7c8e2ef54085a3f74fad2b413f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } } }
I20260923 14:49:40.886880 18852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:40.887176 18851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f9e7c8e2ef54085a3f74fad2b413f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e7c8e2ef54085a3f74fad2b413f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } } }
I20260923 14:49:40.887270 18851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:40.888139 18860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:40.890936 18860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:40.896117 18860 catalog_manager.cc:1347] Generated new cluster ID: 83c63d9aa3084764b47563b1d8cadaf3
I20260923 14:49:40.896179 18860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:40.906373 18860 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:40.906723 18860 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:40.913117 18860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f9e7c8e2ef54085a3f74fad2b413f79: Generated new TSK 0
I20260923 14:49:40.913861 18860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:40.918494 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 18772
2026-09-23T14:49:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:40 dist-test-slave-wjf2 krb5kdc[18872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:40 dist-test-slave-wjf2 krb5kdc[18872](info): set up 2 sockets
Sep 23 14:49:40 dist-test-slave-wjf2 krb5kdc[18872](info): commencing operation
krb5kdc: starting...
W20260923 14:49:42.001312 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:42 dist-test-slave-wjf2 krb5kdc[18872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:42.123600 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:42.231537 18885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:42.231863 18885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:42.231943 18885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:42.235918 18885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:42.236011 18885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:42.236050 18885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:42.236086 18885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:42.236119 18885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:42.241422 18885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:42.242501 18885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:42.243599 18885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:42.252058 18890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:42.252034 18891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:42.252058 18893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:42.252267 18885 server_base.cc:1034] running on GCE node
I20260923 14:49:42.252872 18885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:42.253966 18885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:42.255182 18885 hybrid_clock.cc:648] HybridClock initialized: now 1790174982255166 us; error 24 us; skew 500 ppm
Sep 23 14:49:42 dist-test-slave-wjf2 krb5kdc[18872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:42.261945 18885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:42.262920 18885 webserver.cc:460] Webserver started at http://127.0.0.1:34331/ using document root <none> and password file <none>
I20260923 14:49:42.263536 18885 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:42.263609 18885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:42.263942 18885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:42.266084 18885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "b1906fe1383b477da700a63abccb71b3"
format_stamp: "Formatted at 2026-09-23 14:49:42 on dist-test-slave-wjf2"
server_key: "8a75488d38e7c22277bea9f0dfbb15c3"
server_key_iv: "8fa523aad2941d6a7e9addbab98c9fe9"
server_key_version: "encryptionkey@0"
I20260923 14:49:42.266685 18885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "b1906fe1383b477da700a63abccb71b3"
format_stamp: "Formatted at 2026-09-23 14:49:42 on dist-test-slave-wjf2"
server_key: "8a75488d38e7c22277bea9f0dfbb15c3"
server_key_iv: "8fa523aad2941d6a7e9addbab98c9fe9"
server_key_version: "encryptionkey@0"
I20260923 14:49:42.270533 18885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260923 14:49:42.273056 18900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:42.274223 18885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:49:42.274360 18885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "b1906fe1383b477da700a63abccb71b3"
format_stamp: "Formatted at 2026-09-23 14:49:42 on dist-test-slave-wjf2"
server_key: "8a75488d38e7c22277bea9f0dfbb15c3"
server_key_iv: "8fa523aad2941d6a7e9addbab98c9fe9"
server_key_version: "encryptionkey@0"
I20260923 14:49:42.274466 18885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:42.294265 18885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:42.297559 18885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:42.297773 18885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:42.305423 18885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33169
I20260923 14:49:42.305449 18952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33169 every 8 connection(s)
I20260923 14:49:42.306372 18885 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:42.309139 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 18885
I20260923 14:49:42.309284 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:42.309473 18953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:42.309670 30731 external_mini_cluster.cc:1428] Setting key a05f62a712cde8085d9483daf5913fe9
I20260923 14:49:42.317381 18953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3: Bootstrap starting.
Sep 23 14:49:42 dist-test-slave-wjf2 krb5kdc[18872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 14:49:42.319871 18953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:42.320808 18953 log.cc:824] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:42.323333 18953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3: No bootstrap required, opened a new log
I20260923 14:49:42.325513 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:42.311164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32889 (local address 127.0.0.1:33169)
0923 14:49:42.311608 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:42.311630 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:42.311675 (+ 45us) server_negotiation.cc:407] Connection header received
0923 14:49:42.312523 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:42.312550 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:42.313021 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:42.313427 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:42.314219 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:42.315951 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:42.316853 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:42.317598 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:42.317786 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:42.320301 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:42.320309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:42.320326 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:42.320359 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:42.322625 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:42.323055 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:42.323061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:42.323067 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:42.323213 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:42.323542 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:42.323546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:42.323553 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:42.324027 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:42.324229 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:42.324528 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 14:49:42.324811 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":288,"thread_start_us":120,"threads_started":1}
I20260923 14:49:42.327422 18953 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1906fe1383b477da700a63abccb71b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } }
I20260923 14:49:42.327733 18953 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:42.327853 18953 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1906fe1383b477da700a63abccb71b3, State: Initialized, Role: FOLLOWER
I20260923 14:49:42.328442 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:42.328378 18953 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1906fe1383b477da700a63abccb71b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } }
I20260923 14:49:42.328559 18953 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:42.328632 18953 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:42.328769 18953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:42.329720 18953 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1906fe1383b477da700a63abccb71b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } }
I20260923 14:49:42.330088 18953 leader_election.cc:302] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1906fe1383b477da700a63abccb71b3; no voters:
I20260923 14:49:42.330438 18953 leader_election.cc:288] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:42.330519 18962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:42.330802 18962 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [term 1 LEADER]: Becoming Leader. State: Replica: b1906fe1383b477da700a63abccb71b3, State: Running, Role: LEADER
I20260923 14:49:42.331207 18962 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1906fe1383b477da700a63abccb71b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } }
I20260923 14:49:42.331966 18953 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:42.332888 18965 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1906fe1383b477da700a63abccb71b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1906fe1383b477da700a63abccb71b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } } }
I20260923 14:49:42.333066 18965 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:42.333474 18964 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1906fe1383b477da700a63abccb71b3. Latest consensus state: current_term: 1 leader_uuid: "b1906fe1383b477da700a63abccb71b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1906fe1383b477da700a63abccb71b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33169 } } }
I20260923 14:49:42.333648 18964 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3 [sys.catalog]: This master's current role is: LEADER
W20260923 14:49:42.337219 18978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:49:42.337313 18978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:49:42.337635 18972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:42.338357 18972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:42.340312 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:42.330519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50183 (local address 127.0.0.1:33169)
0923 14:49:42.330657 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:42.330661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:42.330905 (+ 244us) server_negotiation.cc:407] Connection header received
0923 14:49:42.331114 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:42.331118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:42.331164 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:42.331243 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:42.332153 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:42.333084 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:42.334411 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:42.334723 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:42.334827 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:42.336492 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:42.336496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:42.336499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:42.336526 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:42.338419 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:42.338965 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:42.338969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:42.338970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:42.339057 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:42.339408 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:42.339411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:42.339413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:42.339616 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:42.339748 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:42.340055 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 14:49:42.340170 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:49:42.344281 18972 catalog_manager.cc:1347] Generated new cluster ID: 04ede3649e8d43ee9358f625036eab79
I20260923 14:49:42.344342 18972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:42.363161 18972 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:42.364137 18972 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:42.371017 18972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1906fe1383b477da700a63abccb71b3: Generated new TSK 0
I20260923 14:49:42.371641 18972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:42.376359 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 18885
2026-09-23T14:49:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:42 dist-test-slave-wjf2 krb5kdc[18984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:42 dist-test-slave-wjf2 krb5kdc[18984](info): set up 2 sockets
Sep 23 14:49:42 dist-test-slave-wjf2 krb5kdc[18984](info): commencing operation
krb5kdc: starting...
W20260923 14:49:43.413349 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:43 dist-test-slave-wjf2 krb5kdc[18984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:43.538381 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:43.648622 18997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:43.648849 18997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:43.648936 18997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:43.652872 18997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:43.652925 18997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:43.652944 18997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:43.652997 18997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:43.653031 18997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:43.658110 18997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:43.659157 18997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:43.660190 18997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:43.668761 19005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:43.668771 19003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:43.668771 19002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:43.668993 18997 server_base.cc:1034] running on GCE node
I20260923 14:49:43.669620 18997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:43.670642 18997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:43.671861 18997 hybrid_clock.cc:648] HybridClock initialized: now 1790174983671828 us; error 42 us; skew 500 ppm
Sep 23 14:49:43 dist-test-slave-wjf2 krb5kdc[18984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:43.674474 18997 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:43.675292 18997 webserver.cc:460] Webserver started at http://127.0.0.1:45447/ using document root <none> and password file <none>
I20260923 14:49:43.675887 18997 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:43.675954 18997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:43.676136 18997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:43.678134 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "cab523ea6ee4499785ff6bfa25ffa4d8"
format_stamp: "Formatted at 2026-09-23 14:49:43 on dist-test-slave-wjf2"
server_key: "06f4c6ee8506ec9c218e413fe27ed0a9"
server_key_iv: "527f407e8e537c3a0e0e27e1da2345d8"
server_key_version: "encryptionkey@0"
I20260923 14:49:43.678627 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "cab523ea6ee4499785ff6bfa25ffa4d8"
format_stamp: "Formatted at 2026-09-23 14:49:43 on dist-test-slave-wjf2"
server_key: "06f4c6ee8506ec9c218e413fe27ed0a9"
server_key_iv: "527f407e8e537c3a0e0e27e1da2345d8"
server_key_version: "encryptionkey@0"
I20260923 14:49:43.682238 18997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:49:43.684561 19012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:43.685834 18997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:49:43.685946 18997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "cab523ea6ee4499785ff6bfa25ffa4d8"
format_stamp: "Formatted at 2026-09-23 14:49:43 on dist-test-slave-wjf2"
server_key: "06f4c6ee8506ec9c218e413fe27ed0a9"
server_key_iv: "527f407e8e537c3a0e0e27e1da2345d8"
server_key_version: "encryptionkey@0"
I20260923 14:49:43.686033 18997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:43.705552 18997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:43.708963 18997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:43.709127 18997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:43.716665 18997 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45419
I20260923 14:49:43.716718 19064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45419 every 8 connection(s)
I20260923 14:49:43.717720 18997 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:43.720543 19065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:43.724224 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 18997
I20260923 14:49:43.724370 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:43.724653 30731 external_mini_cluster.cc:1428] Setting key 2cdeecc4af2cc6b60ba46b15c854fa83
I20260923 14:49:43.726440 19065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8: Bootstrap starting.
I20260923 14:49:43.728683 19065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:43.729400 19065 log.cc:824] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:43.731129 19065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8: No bootstrap required, opened a new log
I20260923 14:49:43.733367 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:43.726040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55677 (local address 127.0.0.1:45419)
0923 14:49:43.726469 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:43.726484 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:43.726519 (+ 35us) server_negotiation.cc:407] Connection header received
0923 14:49:43.727175 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:43.727191 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:43.727521 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:43.727886 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:43.728295 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:43.729334 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:43.730274 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:43.730741 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:43.730912 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:43.731751 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:43.731764 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:43.731817 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:43.732186 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:43.732245 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:43.732267 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:43.732372 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:43.732501 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 14:49:43.732705 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":123,"threads_started":1}
I20260923 14:49:43.734661 19065 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab523ea6ee4499785ff6bfa25ffa4d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } }
I20260923 14:49:43.734901 19065 raft_consensus.cc:374] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:43.734957 19065 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab523ea6ee4499785ff6bfa25ffa4d8, State: Initialized, Role: FOLLOWER
I20260923 14:49:43.735442 19065 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab523ea6ee4499785ff6bfa25ffa4d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } }
I20260923 14:49:43.735597 19065 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:43.735690 19065 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:43.735841 19065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:43.736162 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:43.736923 19065 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab523ea6ee4499785ff6bfa25ffa4d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } }
I20260923 14:49:43.737279 19065 leader_election.cc:302] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab523ea6ee4499785ff6bfa25ffa4d8; no voters:
I20260923 14:49:43.737612 19065 leader_election.cc:288] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:43.737720 19073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:43.737950 19073 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [term 1 LEADER]: Becoming Leader. State: Replica: cab523ea6ee4499785ff6bfa25ffa4d8, State: Running, Role: LEADER
I20260923 14:49:43.738292 19073 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab523ea6ee4499785ff6bfa25ffa4d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } }
I20260923 14:49:43.738787 19065 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:43.739512 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab523ea6ee4499785ff6bfa25ffa4d8. Latest consensus state: current_term: 1 leader_uuid: "cab523ea6ee4499785ff6bfa25ffa4d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab523ea6ee4499785ff6bfa25ffa4d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } } }
I20260923 14:49:43.739638 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:43.739953 19075 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cab523ea6ee4499785ff6bfa25ffa4d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab523ea6ee4499785ff6bfa25ffa4d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } } }
I20260923 14:49:43.740097 19084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:43.740113 19075 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:43.743100 19084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:43.744630 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:43.738306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56081 (local address 127.0.0.1:45419)
0923 14:49:43.738397 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:43.738401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:43.740178 (+ 1777us) server_negotiation.cc:407] Connection header received
0923 14:49:43.740366 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:43.740372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:43.740425 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:43.740481 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:43.740931 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:43.741682 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:43.742797 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:43.743093 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:43.743201 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:43.744058 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:43.744062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:43.744064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:43.744076 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:43.744103 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:43.744116 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:43.744181 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:43.744464 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 14:49:43.744512 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:49:43.748656 19084 catalog_manager.cc:1347] Generated new cluster ID: e87e417ed2064501a11071669f2698f4
I20260923 14:49:43.748731 19084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:43.764673 19084 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:43.765908 19084 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:43.770534 19084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cab523ea6ee4499785ff6bfa25ffa4d8: Generated new TSK 0
I20260923 14:49:43.771360 19084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:43.777709 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 18997
2026-09-23T14:49:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:43 dist-test-slave-wjf2 krb5kdc[19096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:43 dist-test-slave-wjf2 krb5kdc[19096](info): set up 2 sockets
Sep 23 14:49:43 dist-test-slave-wjf2 krb5kdc[19096](info): commencing operation
krb5kdc: starting...
W20260923 14:49:44.817356 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:44 dist-test-slave-wjf2 krb5kdc[19096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:44.942615 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:45.054973 19109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:45.055297 19109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:45.055342 19109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:45.059723 19109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:45.059870 19109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:45.059897 19109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:45.059916 19109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:45.059932 19109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:45.066131 19109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:45.067414 19109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:45.068614 19109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:45.077979 19115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:45.078076 19114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:45.078017 19117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:45.078365 19109 server_base.cc:1034] running on GCE node
I20260923 14:49:45.079059 19109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:45.080411 19109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:45.081604 19109 hybrid_clock.cc:648] HybridClock initialized: now 1790174985081571 us; error 41 us; skew 500 ppm
Sep 23 14:49:45 dist-test-slave-wjf2 krb5kdc[19096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:45.084613 19109 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:45.085565 19109 webserver.cc:460] Webserver started at http://127.0.0.1:45655/ using document root <none> and password file <none>
I20260923 14:49:45.086161 19109 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:45.086256 19109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:45.086520 19109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:45.088724 19109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "016b990054e2435eb9092a34cbd5a95d"
format_stamp: "Formatted at 2026-09-23 14:49:45 on dist-test-slave-wjf2"
server_key: "363afdf72451930da0ffd5d8a63d4645"
server_key_iv: "3b072511ee7b558f16fbfb4880e95848"
server_key_version: "encryptionkey@0"
I20260923 14:49:45.089314 19109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "016b990054e2435eb9092a34cbd5a95d"
format_stamp: "Formatted at 2026-09-23 14:49:45 on dist-test-slave-wjf2"
server_key: "363afdf72451930da0ffd5d8a63d4645"
server_key_iv: "3b072511ee7b558f16fbfb4880e95848"
server_key_version: "encryptionkey@0"
I20260923 14:49:45.093110 19109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:49:45.095623 19124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:45.097020 19109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:49:45.097191 19109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "016b990054e2435eb9092a34cbd5a95d"
format_stamp: "Formatted at 2026-09-23 14:49:45 on dist-test-slave-wjf2"
server_key: "363afdf72451930da0ffd5d8a63d4645"
server_key_iv: "3b072511ee7b558f16fbfb4880e95848"
server_key_version: "encryptionkey@0"
I20260923 14:49:45.097348 19109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:45.102727 19109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:45.105872 19109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:45.106071 19109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:45.113380 19109 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38835
I20260923 14:49:45.113436 19176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38835 every 8 connection(s)
I20260923 14:49:45.114368 19109 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:45.117323 19177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:45.119191 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 19109
I20260923 14:49:45.119334 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:45.119625 30731 external_mini_cluster.cc:1428] Setting key 1c10d7dd0e7bb9278ad5fff28c176c6f
I20260923 14:49:45.124162 19177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d: Bootstrap starting.
I20260923 14:49:45.125126 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:45.121076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44811 (local address 127.0.0.1:38835)
0923 14:49:45.121498 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:45.121515 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:45.121551 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:49:45.122179 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:45.122196 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:45.122485 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:45.122754 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:45.123687 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:45.123694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:45.123744 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:45.123759 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:45.123870 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:45.123903 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:45.124062 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:45.124192 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 14:49:45.124376 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":135,"threads_started":1}
I20260923 14:49:45.126580 19177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:45.127432 19177 log.cc:824] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:45.128000 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:45.129431 19177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d: No bootstrap required, opened a new log
I20260923 14:49:45.131649 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:45.130013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55235 (local address 127.0.0.1:38835)
0923 14:49:45.130134 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:45.130139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:45.130380 (+ 241us) server_negotiation.cc:407] Connection header received
0923 14:49:45.130489 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:45.130491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:45.130518 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:45.130580 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:45.131244 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:45.131247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:45.131249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:45.131259 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:45.131281 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:45.131306 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:45.131371 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:45.131486 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 14:49:45.131517 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:49:45.133610 19177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016b990054e2435eb9092a34cbd5a95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38835 } }
I20260923 14:49:45.133846 19177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:45.133924 19177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016b990054e2435eb9092a34cbd5a95d, State: Initialized, Role: FOLLOWER
I20260923 14:49:45.134393 19177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016b990054e2435eb9092a34cbd5a95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38835 } }
I20260923 14:49:45.134542 19177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:45.134591 19177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:45.134661 19177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:45.135480 19177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016b990054e2435eb9092a34cbd5a95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38835 } }
I20260923 14:49:45.135811 19177 leader_election.cc:302] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 016b990054e2435eb9092a34cbd5a95d; no voters:
I20260923 14:49:45.136077 19177 leader_election.cc:288] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:45.136195 19187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:45.136416 19187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [term 1 LEADER]: Becoming Leader. State: Replica: 016b990054e2435eb9092a34cbd5a95d, State: Running, Role: LEADER
I20260923 14:49:45.136761 19187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016b990054e2435eb9092a34cbd5a95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38835 } }
I20260923 14:49:45.137392 19177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:45.138438 19188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "016b990054e2435eb9092a34cbd5a95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016b990054e2435eb9092a34cbd5a95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38835 } } }
I20260923 14:49:45.138566 19188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:45.138932 19196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:45.138890 19189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 016b990054e2435eb9092a34cbd5a95d. Latest consensus state: current_term: 1 leader_uuid: "016b990054e2435eb9092a34cbd5a95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016b990054e2435eb9092a34cbd5a95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38835 } } }
I20260923 14:49:45.139083 19189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:45.141782 19196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:45.146704 19196 catalog_manager.cc:1347] Generated new cluster ID: 6458e7fe426a4d258be1b8b5d69d739d
I20260923 14:49:45.146766 19196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:45.158452 19196 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:45.158758 19196 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:45.166857 19196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 016b990054e2435eb9092a34cbd5a95d: Generated new TSK 0
I20260923 14:49:45.167585 19196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:45.178822 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 19109
2026-09-23T14:49:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:45 dist-test-slave-wjf2 krb5kdc[19208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:45 dist-test-slave-wjf2 krb5kdc[19208](info): set up 2 sockets
Sep 23 14:49:45 dist-test-slave-wjf2 krb5kdc[19208](info): commencing operation
krb5kdc: starting...
W20260923 14:49:46.233443 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:46 dist-test-slave-wjf2 krb5kdc[19208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:46.358695 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:46.475276 19221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:46.475557 19221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:46.475629 19221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:46.479667 19221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:46.479739 19221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:46.479771 19221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:46.479838 19221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:46.479876 19221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:46.485337 19221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:46.486398 19221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:46.487485 19221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:46.496258 19227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:46.496266 19226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:46.496300 19221 server_base.cc:1034] running on GCE node
W20260923 14:49:46.496264 19229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:46.497045 19221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:46.498129 19221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:46.499331 19221 hybrid_clock.cc:648] HybridClock initialized: now 1790174986499317 us; error 27 us; skew 500 ppm
Sep 23 14:49:46 dist-test-slave-wjf2 krb5kdc[19208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:46.502146 19221 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:46.503041 19221 webserver.cc:460] Webserver started at http://127.0.0.1:45559/ using document root <none> and password file <none>
I20260923 14:49:46.503610 19221 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:46.503698 19221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:46.503964 19221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:46.506029 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "28edba7455b04e2c821729bcb1e3f2ca"
format_stamp: "Formatted at 2026-09-23 14:49:46 on dist-test-slave-wjf2"
server_key: "c575f683d56643a74cf63b2aaea29ce7"
server_key_iv: "84f791e5e88847377aab5178185fc1bf"
server_key_version: "encryptionkey@0"
I20260923 14:49:46.506593 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "28edba7455b04e2c821729bcb1e3f2ca"
format_stamp: "Formatted at 2026-09-23 14:49:46 on dist-test-slave-wjf2"
server_key: "c575f683d56643a74cf63b2aaea29ce7"
server_key_iv: "84f791e5e88847377aab5178185fc1bf"
server_key_version: "encryptionkey@0"
I20260923 14:49:46.510245 19221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260923 14:49:46.512679 19236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:46.513906 19221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:46.514039 19221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "28edba7455b04e2c821729bcb1e3f2ca"
format_stamp: "Formatted at 2026-09-23 14:49:46 on dist-test-slave-wjf2"
server_key: "c575f683d56643a74cf63b2aaea29ce7"
server_key_iv: "84f791e5e88847377aab5178185fc1bf"
server_key_version: "encryptionkey@0"
I20260923 14:49:46.514143 19221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:46.527948 19221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:46.531033 19221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:46.531239 19221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:46.538442 19221 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40613
I20260923 14:49:46.538458 19288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40613 every 8 connection(s)
I20260923 14:49:46.539454 19221 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:46.542457 19289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:46.545140 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 19221
I20260923 14:49:46.545293 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:46.545600 30731 external_mini_cluster.cc:1428] Setting key ef5fdca9ff4c698d66dc11008488b6cd
I20260923 14:49:46.548971 19289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca: Bootstrap starting.
I20260923 14:49:46.551192 19289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:46.551990 19289 log.cc:824] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:46.554390 19289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca: No bootstrap required, opened a new log
Sep 23 14:49:46 dist-test-slave-wjf2 krb5kdc[19208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 14:49:46.559109 19289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28edba7455b04e2c821729bcb1e3f2ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260923 14:49:46.559417 19289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:46.559546 19289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28edba7455b04e2c821729bcb1e3f2ca, State: Initialized, Role: FOLLOWER
I20260923 14:49:46.560113 19289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28edba7455b04e2c821729bcb1e3f2ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260923 14:49:46.560286 19289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:46.560377 19289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:46.560478 19289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:46.560806 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:46.547144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:49989 (local address 127.0.0.1:40613)
0923 14:49:46.547620 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:46.547635 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:46.547671 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:49:46.548407 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:46.548425 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:46.548785 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:46.549124 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:46.549916 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:46.550900 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:46.551895 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:46.552472 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:46.552641 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:46.555504 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:46.555510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:46.555527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:46.555558 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:46.557610 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:46.558132 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:46.558138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:46.558146 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:46.558300 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:46.558618 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:46.558621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:46.558623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:46.559119 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:46.559376 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:46.559707 (+ 331us) server_negotiation.cc:299] Negotiation successful
0923 14:49:46.559997 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":137,"threads_started":1}
I20260923 14:49:46.561527 19289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28edba7455b04e2c821729bcb1e3f2ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260923 14:49:46.561896 19289 leader_election.cc:302] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28edba7455b04e2c821729bcb1e3f2ca; no voters:
I20260923 14:49:46.562249 19289 leader_election.cc:288] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:46.562358 19294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:46.562530 19294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [term 1 LEADER]: Becoming Leader. State: Replica: 28edba7455b04e2c821729bcb1e3f2ca, State: Running, Role: LEADER
I20260923 14:49:46.562896 19294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28edba7455b04e2c821729bcb1e3f2ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260923 14:49:46.563616 19289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:46.564452 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:46.565078 19295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28edba7455b04e2c821729bcb1e3f2ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28edba7455b04e2c821729bcb1e3f2ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } } }
I20260923 14:49:46.565511 19295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:46.565390 19296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28edba7455b04e2c821729bcb1e3f2ca. Latest consensus state: current_term: 1 leader_uuid: "28edba7455b04e2c821729bcb1e3f2ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28edba7455b04e2c821729bcb1e3f2ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } } }
I20260923 14:49:46.565826 19296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca [sys.catalog]: This master's current role is: LEADER
W20260923 14:49:46.569069 19309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:49:46.569211 19309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:49:46.569324 19308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:46.570052 19308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:46.575933 19308 catalog_manager.cc:1347] Generated new cluster ID: 322c201d51f94d2f89dc503bf83b7cbb
I20260923 14:49:46.576047 19308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:46.577929 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:46.568443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:35611 (local address 127.0.0.1:40613)
0923 14:49:46.568559 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:46.568563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:46.568937 (+ 374us) server_negotiation.cc:407] Connection header received
0923 14:49:46.569066 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:46.569069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:46.569127 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:46.569214 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:46.570073 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:46.570841 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:46.572079 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:46.572391 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:46.572499 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:46.574302 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:46.574308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:46.574315 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:46.574343 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:46.575964 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:46.576562 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:46.576566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:46.576567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:46.576649 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:46.577043 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:46.577047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:46.577048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:46.577235 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:46.577384 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:46.577666 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 14:49:46.577781 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:49:46.604331 19308 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:46.605291 19308 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:46.610984 19308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28edba7455b04e2c821729bcb1e3f2ca: Generated new TSK 0
I20260923 14:49:46.611855 19308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:46.615314 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 19221
W20260923 14:49:46.621994 19206 connection.cc:535] client connection to 127.0.0.1:40613 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40613): Connection reset by peer (error 104)
2026-09-23T14:49:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:46 dist-test-slave-wjf2 krb5kdc[19320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:46 dist-test-slave-wjf2 krb5kdc[19320](info): set up 2 sockets
Sep 23 14:49:46 dist-test-slave-wjf2 krb5kdc[19320](info): commencing operation
krb5kdc: starting...
W20260923 14:49:47.657250 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:47 dist-test-slave-wjf2 krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:47.779245 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:47.894156 19333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:47.894418 19333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:47.894460 19333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:47.898563 19333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:47.898628 19333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:47.898648 19333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:47.898665 19333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:47.898680 19333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:47.904963 19333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:47.906282 19333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:47.907451 19333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:47.916796 19339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:47.916762 19338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:47.916802 19341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:47.917052 19333 server_base.cc:1034] running on GCE node
I20260923 14:49:47.917801 19333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:47.919036 19333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:47.920265 19333 hybrid_clock.cc:648] HybridClock initialized: now 1790174987920244 us; error 34 us; skew 500 ppm
Sep 23 14:49:47 dist-test-slave-wjf2 krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:47.927438 19333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:47.928414 19333 webserver.cc:460] Webserver started at http://127.0.0.1:40363/ using document root <none> and password file <none>
I20260923 14:49:47.928968 19333 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:47.929025 19333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:47.929188 19333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:47.931630 19333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "a35c980ff90e49d6893518b8d6b0ce88"
format_stamp: "Formatted at 2026-09-23 14:49:47 on dist-test-slave-wjf2"
server_key: "da00299d2a53221265a4b7dd9409c8a1"
server_key_iv: "83edef7c97c5c632c4eebc9d38d2b8d5"
server_key_version: "encryptionkey@0"
I20260923 14:49:47.932210 19333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "a35c980ff90e49d6893518b8d6b0ce88"
format_stamp: "Formatted at 2026-09-23 14:49:47 on dist-test-slave-wjf2"
server_key: "da00299d2a53221265a4b7dd9409c8a1"
server_key_iv: "83edef7c97c5c632c4eebc9d38d2b8d5"
server_key_version: "encryptionkey@0"
I20260923 14:49:47.935987 19333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:49:47.938450 19348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:47.939731 19333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:49:47.939889 19333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "a35c980ff90e49d6893518b8d6b0ce88"
format_stamp: "Formatted at 2026-09-23 14:49:47 on dist-test-slave-wjf2"
server_key: "da00299d2a53221265a4b7dd9409c8a1"
server_key_iv: "83edef7c97c5c632c4eebc9d38d2b8d5"
server_key_version: "encryptionkey@0"
I20260923 14:49:47.939978 19333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:47.964179 19333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:47.967657 19333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:47.968014 19333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:47.975095 19333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39207
I20260923 14:49:47.975112 19400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39207 every 8 connection(s)
I20260923 14:49:47.976167 19333 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:47.979142 19401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:47.985193 19401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88: Bootstrap starting.
I20260923 14:49:47.986001 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 19333
I20260923 14:49:47.986171 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:47.986572 30731 external_mini_cluster.cc:1428] Setting key f02a03b7007908384f8e9df7be23e28b
I20260923 14:49:47.988766 19401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:47.989591 19401 log.cc:824] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:47.991664 19401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88: No bootstrap required, opened a new log
I20260923 14:49:47.995903 19401 raft_consensus.cc:348] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35c980ff90e49d6893518b8d6b0ce88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } }
I20260923 14:49:47.996138 19401 raft_consensus.cc:374] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:47.996243 19401 raft_consensus.cc:729] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35c980ff90e49d6893518b8d6b0ce88, State: Initialized, Role: FOLLOWER
I20260923 14:49:47.996531 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:47.988168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:37389 (local address 127.0.0.1:39207)
0923 14:49:47.988643 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:47.988658 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:47.988694 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:49:47.989440 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:47.989459 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:47.989890 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:47.990239 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:47.990674 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:47.991709 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:47.992735 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:47.993233 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:47.993428 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:47.994511 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:47.994517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:47.994531 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:47.994962 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:47.995020 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:47.995045 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:47.995199 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:47.995406 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 14:49:47.995550 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":140,"threads_started":1}
I20260923 14:49:47.996832 19401 consensus_queue.cc:260] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35c980ff90e49d6893518b8d6b0ce88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } }
I20260923 14:49:47.996979 19401 raft_consensus.cc:388] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:47.997105 19401 raft_consensus.cc:482] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:47.997241 19401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:47.998186 19401 raft_consensus.cc:504] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35c980ff90e49d6893518b8d6b0ce88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } }
I20260923 14:49:47.998618 19401 leader_election.cc:302] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a35c980ff90e49d6893518b8d6b0ce88; no voters:
I20260923 14:49:47.999020 19401 leader_election.cc:288] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:47.999096 19406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:47.999383 19406 raft_consensus.cc:686] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [term 1 LEADER]: Becoming Leader. State: Replica: a35c980ff90e49d6893518b8d6b0ce88, State: Running, Role: LEADER
I20260923 14:49:47.999771 19406 consensus_queue.cc:237] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35c980ff90e49d6893518b8d6b0ce88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } }
I20260923 14:49:47.999861 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:48.000537 19401 sys_catalog.cc:564] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:48.001530 19407 sys_catalog.cc:455] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a35c980ff90e49d6893518b8d6b0ce88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35c980ff90e49d6893518b8d6b0ce88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } } }
I20260923 14:49:48.001691 19407 sys_catalog.cc:458] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:48.002924 19418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:48.001963 19408 sys_catalog.cc:455] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a35c980ff90e49d6893518b8d6b0ce88. Latest consensus state: current_term: 1 leader_uuid: "a35c980ff90e49d6893518b8d6b0ce88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35c980ff90e49d6893518b8d6b0ce88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } } }
I20260923 14:49:48.003472 19408 sys_catalog.cc:458] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:48.006704 19418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:48.015334 19418 catalog_manager.cc:1347] Generated new cluster ID: 293228063d294cb1bd9ba456e25aad7f
I20260923 14:49:48.015355 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:48.005806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:46969 (local address 127.0.0.1:39207)
0923 14:49:48.005945 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:48.005951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:48.006764 (+ 813us) server_negotiation.cc:407] Connection header received
0923 14:49:48.006989 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:48.006993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:48.007044 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:48.007101 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:48.007639 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:48.008752 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:48.013061 (+ 4309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:48.013333 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:48.013460 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:48.014745 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:48.014752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:48.014755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:48.014772 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:48.014801 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:48.014816 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:48.014888 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:48.015157 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 14:49:48.015199 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:49:48.015420 19418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:48.065654 19418 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:48.066799 19418 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:48.077699 19418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a35c980ff90e49d6893518b8d6b0ce88: Generated new TSK 0
I20260923 14:49:48.078567 19418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:48.096091 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 19333
W20260923 14:49:48.101907 19318 connection.cc:535] client connection to 127.0.0.1:39207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39207): Connection reset by peer (error 104)
2026-09-23T14:49:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1480 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:48 dist-test-slave-wjf2 krb5kdc[19432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:48 dist-test-slave-wjf2 krb5kdc[19432](info): set up 2 sockets
Sep 23 14:49:48 dist-test-slave-wjf2 krb5kdc[19432](info): commencing operation
krb5kdc: starting...
W20260923 14:49:49.146516 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:49 dist-test-slave-wjf2 krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:49.279441 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:49.400074 19445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:49.400463 19445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:49.400589 19445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:49.405030 19445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:49.405088 19445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:49.405108 19445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:49.405125 19445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:49.405140 19445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:49.410524 19445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:49.411646 19445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:49.412717 19445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:49.421996 19450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:49.422008 19453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:49.422008 19451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:49.422052 19445 server_base.cc:1034] running on GCE node
I20260923 14:49:49.422708 19445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:49.423843 19445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:49.425021 19445 hybrid_clock.cc:648] HybridClock initialized: now 1790174989424991 us; error 42 us; skew 500 ppm
Sep 23 14:49:49 dist-test-slave-wjf2 krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:49.427896 19445 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:49.428889 19445 webserver.cc:460] Webserver started at http://127.0.0.1:44321/ using document root <none> and password file <none>
I20260923 14:49:49.429517 19445 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:49.429582 19445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:49.429812 19445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:49.432081 19445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "c21c369a8c5d4095bdd031dc45e68b11"
format_stamp: "Formatted at 2026-09-23 14:49:49 on dist-test-slave-wjf2"
server_key: "0913f68c425761fbe3a109d856b1a211"
server_key_iv: "a449ae7a235aa37b9588ed920054b500"
server_key_version: "encryptionkey@0"
I20260923 14:49:49.432703 19445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "c21c369a8c5d4095bdd031dc45e68b11"
format_stamp: "Formatted at 2026-09-23 14:49:49 on dist-test-slave-wjf2"
server_key: "0913f68c425761fbe3a109d856b1a211"
server_key_iv: "a449ae7a235aa37b9588ed920054b500"
server_key_version: "encryptionkey@0"
I20260923 14:49:49.436779 19445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:49:49.439258 19460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:49.440658 19445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:49.440834 19445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "c21c369a8c5d4095bdd031dc45e68b11"
format_stamp: "Formatted at 2026-09-23 14:49:49 on dist-test-slave-wjf2"
server_key: "0913f68c425761fbe3a109d856b1a211"
server_key_iv: "a449ae7a235aa37b9588ed920054b500"
server_key_version: "encryptionkey@0"
I20260923 14:49:49.440946 19445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:49.445832 19445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:49.449069 19445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:49.449290 19445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:49.456820 19445 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34669
I20260923 14:49:49.456835 19512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34669 every 8 connection(s)
I20260923 14:49:49.457823 19445 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:49.460952 19513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:49.466329 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 19445
I20260923 14:49:49.466503 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:49.466883 30731 external_mini_cluster.cc:1428] Setting key 2339dca6687d4bd1c98b23f27c9b883b
I20260923 14:49:49.467195 19513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11: Bootstrap starting.
I20260923 14:49:49.469982 19513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:49.470932 19513 log.cc:824] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:49.472441 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:49.468412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:46223 (local address 127.0.0.1:34669)
0923 14:49:49.468871 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:49.468893 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:49.468931 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:49:49.469576 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:49.469593 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:49.469909 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:49.470307 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:49.471156 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:49.471160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:49.471173 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:49.471183 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:49.471236 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:49.471257 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:49.471375 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:49.471507 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 14:49:49.471677 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":149,"threads_started":1}
I20260923 14:49:49.473155 19513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11: No bootstrap required, opened a new log
I20260923 14:49:49.475135 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:49.477670 19513 raft_consensus.cc:348] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21c369a8c5d4095bdd031dc45e68b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } }
I20260923 14:49:49.478013 19513 raft_consensus.cc:374] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:49.478091 19513 raft_consensus.cc:729] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c21c369a8c5d4095bdd031dc45e68b11, State: Initialized, Role: FOLLOWER
I20260923 14:49:49.478585 19513 consensus_queue.cc:260] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21c369a8c5d4095bdd031dc45e68b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } }
I20260923 14:49:49.478785 19513 raft_consensus.cc:388] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:49.478871 19513 raft_consensus.cc:482] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:49.478991 19513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:49.479470 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:49.477633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:33979 (local address 127.0.0.1:34669)
0923 14:49:49.477750 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:49.477756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:49.477940 (+ 184us) server_negotiation.cc:407] Connection header received
0923 14:49:49.478117 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:49.478122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:49.478162 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:49.478259 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:49.479055 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:49.479060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:49.479064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:49.479079 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:49.479112 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:49.479128 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:49.479245 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:49.479306 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 14:49:49.479350 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:49:49.480254 19513 raft_consensus.cc:504] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21c369a8c5d4095bdd031dc45e68b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } }
I20260923 14:49:49.480611 19513 leader_election.cc:302] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c21c369a8c5d4095bdd031dc45e68b11; no voters:
I20260923 14:49:49.480953 19513 leader_election.cc:288] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:49.481065 19523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:49.481288 19523 raft_consensus.cc:686] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [term 1 LEADER]: Becoming Leader. State: Replica: c21c369a8c5d4095bdd031dc45e68b11, State: Running, Role: LEADER
I20260923 14:49:49.481637 19523 consensus_queue.cc:237] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21c369a8c5d4095bdd031dc45e68b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } }
I20260923 14:49:49.482240 19513 sys_catalog.cc:564] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:49.483454 19525 sys_catalog.cc:455] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c21c369a8c5d4095bdd031dc45e68b11. Latest consensus state: current_term: 1 leader_uuid: "c21c369a8c5d4095bdd031dc45e68b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21c369a8c5d4095bdd031dc45e68b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } } }
I20260923 14:49:49.483444 19524 sys_catalog.cc:455] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c21c369a8c5d4095bdd031dc45e68b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21c369a8c5d4095bdd031dc45e68b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } } }
I20260923 14:49:49.483598 19525 sys_catalog.cc:458] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:49.483630 19524 sys_catalog.cc:458] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:49.484002 19532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:49.487165 19532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:49.492933 19532 catalog_manager.cc:1347] Generated new cluster ID: 41a87cea259a46d2b05e135fb19d0602
I20260923 14:49:49.492996 19532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:49.514681 19532 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:49.515040 19532 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:49.521382 19532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c21c369a8c5d4095bdd031dc45e68b11: Generated new TSK 0
I20260923 14:49:49.522096 19532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:49.529791 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 19445
2026-09-23T14:49:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:49 dist-test-slave-wjf2 krb5kdc[19544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:49 dist-test-slave-wjf2 krb5kdc[19544](info): set up 2 sockets
Sep 23 14:49:49 dist-test-slave-wjf2 krb5kdc[19544](info): commencing operation
krb5kdc: starting...
W20260923 14:49:50.557333 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:50 dist-test-slave-wjf2 krb5kdc[19544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:50.682833 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:50.794705 19557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:50.794947 19557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:50.794986 19557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:50.799099 19557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:50.799151 19557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:50.799170 19557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:50.799186 19557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:50.799230 19557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:50.804437 19557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:50.805446 19557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:50.806488 19557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:50.815281 19563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:50.815276 19562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:50.815281 19565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:50.815650 19557 server_base.cc:1034] running on GCE node
I20260923 14:49:50.816282 19557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:50.817456 19557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:50.818671 19557 hybrid_clock.cc:648] HybridClock initialized: now 1790174990818646 us; error 44 us; skew 500 ppm
Sep 23 14:49:50 dist-test-slave-wjf2 krb5kdc[19544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:50.821686 19557 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:50.822605 19557 webserver.cc:460] Webserver started at http://127.0.0.1:44057/ using document root <none> and password file <none>
I20260923 14:49:50.823187 19557 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:50.823282 19557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:50.823518 19557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:50.825623 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "06b576f7524f4c8299a8fe4cd66fa041"
format_stamp: "Formatted at 2026-09-23 14:49:50 on dist-test-slave-wjf2"
server_key: "f6bfb496a8aeec89ee9450126e485781"
server_key_iv: "9abb3349fe049a7be163afb3cd3986de"
server_key_version: "encryptionkey@0"
I20260923 14:49:50.826203 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "06b576f7524f4c8299a8fe4cd66fa041"
format_stamp: "Formatted at 2026-09-23 14:49:50 on dist-test-slave-wjf2"
server_key: "f6bfb496a8aeec89ee9450126e485781"
server_key_iv: "9abb3349fe049a7be163afb3cd3986de"
server_key_version: "encryptionkey@0"
I20260923 14:49:50.830084 19557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:49:50.832563 19572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:50.833881 19557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:50.834024 19557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "06b576f7524f4c8299a8fe4cd66fa041"
format_stamp: "Formatted at 2026-09-23 14:49:50 on dist-test-slave-wjf2"
server_key: "f6bfb496a8aeec89ee9450126e485781"
server_key_iv: "9abb3349fe049a7be163afb3cd3986de"
server_key_version: "encryptionkey@0"
I20260923 14:49:50.834136 19557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:50.851722 19557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:50.855038 19557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:50.855252 19557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:50.862313 19557 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39989
I20260923 14:49:50.862388 19624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39989 every 8 connection(s)
I20260923 14:49:50.863289 19557 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:50.866277 19625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:50.869042 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 19557
I20260923 14:49:50.869181 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:50.869472 30731 external_mini_cluster.cc:1428] Setting key dc959ebc8284c6a3c4be7a3844627dab
I20260923 14:49:50.872969 19625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041: Bootstrap starting.
I20260923 14:49:50.875860 19625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:50.876760 19625 log.cc:824] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:50.879429 19625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041: No bootstrap required, opened a new log
Sep 23 14:49:50 dist-test-slave-wjf2 krb5kdc[19544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 14:49:50.883245 19625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b576f7524f4c8299a8fe4cd66fa041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39989 } }
I20260923 14:49:50.883500 19625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:50.883597 19625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b576f7524f4c8299a8fe4cd66fa041, State: Initialized, Role: FOLLOWER
I20260923 14:49:50.884110 19625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b576f7524f4c8299a8fe4cd66fa041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39989 } }
I20260923 14:49:50.884270 19625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:50.884343 19625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:50.884465 19625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:50.885517 19625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b576f7524f4c8299a8fe4cd66fa041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39989 } }
I20260923 14:49:50.885782 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:50.870893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:54665 (local address 127.0.0.1:39989)
0923 14:49:50.871333 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:50.871350 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:50.871388 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:49:50.872147 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:50.872167 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:50.872526 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:50.872840 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:50.873819 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:50.874853 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:50.876236 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:50.876760 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:50.876961 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:50.880288 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:50.880295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:50.880310 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:50.880340 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:50.882373 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:50.882950 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:50.882955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:50.882961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:50.883076 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:50.883455 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:50.883466 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:50.883467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:50.884121 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:50.884383 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:50.884703 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 14:49:50.884948 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":151,"threads_started":1}
I20260923 14:49:50.885921 19625 leader_election.cc:302] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06b576f7524f4c8299a8fe4cd66fa041; no voters:
I20260923 14:49:50.886308 19625 leader_election.cc:288] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:50.886448 19631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:50.886700 19631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [term 1 LEADER]: Becoming Leader. State: Replica: 06b576f7524f4c8299a8fe4cd66fa041, State: Running, Role: LEADER
I20260923 14:49:50.887073 19631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b576f7524f4c8299a8fe4cd66fa041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39989 } }
I20260923 14:49:50.887601 19625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:50.888679 19631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06b576f7524f4c8299a8fe4cd66fa041. Latest consensus state: current_term: 1 leader_uuid: "06b576f7524f4c8299a8fe4cd66fa041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b576f7524f4c8299a8fe4cd66fa041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39989 } } }
I20260923 14:49:50.888866 19631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:50.889482 19633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06b576f7524f4c8299a8fe4cd66fa041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b576f7524f4c8299a8fe4cd66fa041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39989 } } }
I20260923 14:49:50.889601 19633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:50.889922 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 14:49:50.892765 19646 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:49:50.892899 19646 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:49:50.893002 19640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:50.893968 19640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:50.900426 19640 catalog_manager.cc:1347] Generated new cluster ID: 37eb3c9dc7dc4642ac0f6787c591d911
I20260923 14:49:50.900570 19640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:50.901825 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:50.892588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:55257 (local address 127.0.0.1:39989)
0923 14:49:50.892699 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:50.892703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:50.892740 (+ 37us) server_negotiation.cc:407] Connection header received
0923 14:49:50.892884 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:50.892889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:50.892945 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:50.893030 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:50.893992 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:50.894869 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:50.896000 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:50.896262 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:50.896378 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:50.897824 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:50.897828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:50.897831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:50.897855 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:50.899629 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:50.900222 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:50.900226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:50.900228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:50.900337 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:50.900759 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:50.900763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:50.900765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:50.901018 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:50.901212 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:50.901529 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 14:49:50.901655 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:49:50.910852 19640 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:50.911906 19640 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:50.920671 19640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06b576f7524f4c8299a8fe4cd66fa041: Generated new TSK 0
I20260923 14:49:50.921402 19640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:50.933089 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 19557
W20260923 14:49:50.939673 19542 connection.cc:535] client connection to 127.0.0.1:39989 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39989): Connection reset by peer (error 104)
2026-09-23T14:49:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:50 dist-test-slave-wjf2 krb5kdc[19657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:50 dist-test-slave-wjf2 krb5kdc[19657](info): set up 2 sockets
Sep 23 14:49:50 dist-test-slave-wjf2 krb5kdc[19657](info): commencing operation
krb5kdc: starting...
W20260923 14:49:51.998041 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:52 dist-test-slave-wjf2 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:52.124622 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:52.237339 19670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:52.237617 19670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:52.237661 19670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:52.242013 19670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:52.242080 19670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:52.242100 19670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:52.242117 19670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:52.242133 19670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:52.247560 19670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:52.248715 19670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:52.249842 19670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:52.258574 19676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:52.258610 19678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:52.258611 19675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:52.258586 19670 server_base.cc:1034] running on GCE node
I20260923 14:49:52.259485 19670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:52.260835 19670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:52.262071 19670 hybrid_clock.cc:648] HybridClock initialized: now 1790174992262047 us; error 37 us; skew 500 ppm
Sep 23 14:49:52 dist-test-slave-wjf2 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:52.265388 19670 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:52.266373 19670 webserver.cc:460] Webserver started at http://127.0.0.1:38055/ using document root <none> and password file <none>
I20260923 14:49:52.267019 19670 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:52.267089 19670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:52.267350 19670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:52.269605 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "b9ef6d4bc43c47458b16082a82b0836d"
format_stamp: "Formatted at 2026-09-23 14:49:52 on dist-test-slave-wjf2"
server_key: "73a8a079c8b481d8a54cae23c9e08af5"
server_key_iv: "9c71b53aa87fc3a8c7ed280ef0b11c90"
server_key_version: "encryptionkey@0"
I20260923 14:49:52.270200 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "b9ef6d4bc43c47458b16082a82b0836d"
format_stamp: "Formatted at 2026-09-23 14:49:52 on dist-test-slave-wjf2"
server_key: "73a8a079c8b481d8a54cae23c9e08af5"
server_key_iv: "9c71b53aa87fc3a8c7ed280ef0b11c90"
server_key_version: "encryptionkey@0"
I20260923 14:49:52.274585 19670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 14:49:52.277851 19685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:52.279356 19670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:49:52.279541 19670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "b9ef6d4bc43c47458b16082a82b0836d"
format_stamp: "Formatted at 2026-09-23 14:49:52 on dist-test-slave-wjf2"
server_key: "73a8a079c8b481d8a54cae23c9e08af5"
server_key_iv: "9c71b53aa87fc3a8c7ed280ef0b11c90"
server_key_version: "encryptionkey@0"
I20260923 14:49:52.279662 19670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:52.304077 19670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:52.307402 19670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:52.307631 19670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:52.315616 19670 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38437
I20260923 14:49:52.315642 19737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38437 every 8 connection(s)
I20260923 14:49:52.316709 19670 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:52.319685 19738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:52.321115 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 19670
I20260923 14:49:52.321259 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:52.321589 30731 external_mini_cluster.cc:1428] Setting key 59828a53e29eabf28f668409e3caa0df
I20260923 14:49:52.326370 19738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d: Bootstrap starting.
I20260923 14:49:52.328904 19738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:52.329849 19738 log.cc:824] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:52.331629 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:52.323157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:36429 (local address 127.0.0.1:38437)
0923 14:49:52.323660 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:52.323677 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:52.323716 (+ 39us) server_negotiation.cc:407] Connection header received
0923 14:49:52.324531 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:52.324551 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:52.324945 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:52.325337 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:52.325779 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:52.326926 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:52.327993 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:52.328541 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:52.328733 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:52.329754 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:52.329760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:52.329775 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:52.330245 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:52.330302 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:52.330328 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:52.330490 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:52.330690 (+ 200us) server_negotiation.cc:299] Negotiation successful
0923 14:49:52.330862 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":157,"threads_started":1}
I20260923 14:49:52.332048 19738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d: No bootstrap required, opened a new log
I20260923 14:49:52.334918 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:52.336633 19738 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef6d4bc43c47458b16082a82b0836d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260923 14:49:52.336902 19738 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:52.336987 19738 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ef6d4bc43c47458b16082a82b0836d, State: Initialized, Role: FOLLOWER
I20260923 14:49:52.337530 19738 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef6d4bc43c47458b16082a82b0836d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260923 14:49:52.337733 19738 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:52.337836 19738 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:52.337961 19738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:52.339134 19738 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef6d4bc43c47458b16082a82b0836d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260923 14:49:52.339522 19738 leader_election.cc:302] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ef6d4bc43c47458b16082a82b0836d; no voters:
I20260923 14:49:52.339902 19738 leader_election.cc:288] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:52.340040 19748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:52.340270 19748 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [term 1 LEADER]: Becoming Leader. State: Replica: b9ef6d4bc43c47458b16082a82b0836d, State: Running, Role: LEADER
I20260923 14:49:52.340718 19748 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef6d4bc43c47458b16082a82b0836d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260923 14:49:52.341369 19738 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:52.342607 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:52.337131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:54697 (local address 127.0.0.1:38437)
0923 14:49:52.337280 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:52.337285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:52.337549 (+ 264us) server_negotiation.cc:407] Connection header received
0923 14:49:52.337635 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:52.337638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:52.337694 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:52.337788 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:52.338289 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:52.338846 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:52.340102 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:52.340490 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:52.340601 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:52.341809 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:52.341813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:52.341816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:52.341830 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:52.341861 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:52.341875 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:52.341966 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:52.342381 (+ 415us) server_negotiation.cc:299] Negotiation successful
0923 14:49:52.342426 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:49:52.342645 19749 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ef6d4bc43c47458b16082a82b0836d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef6d4bc43c47458b16082a82b0836d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } } }
I20260923 14:49:52.342777 19749 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:52.344287 19750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ef6d4bc43c47458b16082a82b0836d. Latest consensus state: current_term: 1 leader_uuid: "b9ef6d4bc43c47458b16082a82b0836d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef6d4bc43c47458b16082a82b0836d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } } }
I20260923 14:49:52.344414 19750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d [sys.catalog]: This master's current role is: LEADER
W20260923 14:49:52.347117 19763 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:49:52.347188 19763 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:49:52.347273 19757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:52.348132 19757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:52.354002 19757 catalog_manager.cc:1347] Generated new cluster ID: ad1f498bc0d0489e81028d7dd901027d
I20260923 14:49:52.354069 19757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:52.366156 19757 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:52.367107 19757 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:52.372901 19757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ef6d4bc43c47458b16082a82b0836d: Generated new TSK 0
I20260923 14:49:52.373765 19757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:52.377592 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 19670
W20260923 14:49:52.383673 19655 connection.cc:535] client connection to 127.0.0.1:38437 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38437): Connection reset by peer (error 104)
2026-09-23T14:49:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:52 dist-test-slave-wjf2 krb5kdc[19769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:52 dist-test-slave-wjf2 krb5kdc[19769](info): set up 2 sockets
Sep 23 14:49:52 dist-test-slave-wjf2 krb5kdc[19769](info): commencing operation
krb5kdc: starting...
W20260923 14:49:53.417367 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:53 dist-test-slave-wjf2 krb5kdc[19769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:49:53.544361 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:53.657253 19782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:53.657557 19782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:53.657632 19782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:53.661648 19782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:53.661737 19782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:53.661777 19782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:53.661835 19782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:53.661881 19782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:53.667171 19782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:53.668267 19782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:53.669291 19782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:53.677691 19790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:53.677723 19782 server_base.cc:1034] running on GCE node
W20260923 14:49:53.677683 19788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:53.677686 19787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:53.678483 19782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:53.679579 19782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:53.680761 19782 hybrid_clock.cc:648] HybridClock initialized: now 1790174993680757 us; error 55 us; skew 500 ppm
Sep 23 14:49:53 dist-test-slave-wjf2 krb5kdc[19769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:53.688239 19782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:49:53.689205 19782 webserver.cc:460] Webserver started at http://127.0.0.1:40671/ using document root <none> and password file <none>
I20260923 14:49:53.689823 19782 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:53.689896 19782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:53.690136 19782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:53.692260 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "bc4b0f6eca6244bf8ac5e72a1c67bdd7"
format_stamp: "Formatted at 2026-09-23 14:49:53 on dist-test-slave-wjf2"
server_key: "94eb01ba9c98dcc8ece23ae21e59591d"
server_key_iv: "8380bc174a049cf80cb2c3bf5df6d4a4"
server_key_version: "encryptionkey@0"
I20260923 14:49:53.692833 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "bc4b0f6eca6244bf8ac5e72a1c67bdd7"
format_stamp: "Formatted at 2026-09-23 14:49:53 on dist-test-slave-wjf2"
server_key: "94eb01ba9c98dcc8ece23ae21e59591d"
server_key_iv: "8380bc174a049cf80cb2c3bf5df6d4a4"
server_key_version: "encryptionkey@0"
I20260923 14:49:53.696549 19782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:49:53.698913 19797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:53.700202 19782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:53.700323 19782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "bc4b0f6eca6244bf8ac5e72a1c67bdd7"
format_stamp: "Formatted at 2026-09-23 14:49:53 on dist-test-slave-wjf2"
server_key: "94eb01ba9c98dcc8ece23ae21e59591d"
server_key_iv: "8380bc174a049cf80cb2c3bf5df6d4a4"
server_key_version: "encryptionkey@0"
I20260923 14:49:53.700428 19782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:53.704941 19782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:53.708266 19782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:53.708461 19782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:53.715404 19782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39225
I20260923 14:49:53.715433 19849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39225 every 8 connection(s)
I20260923 14:49:53.716375 19782 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:53.719050 19850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:53.720331 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 19782
I20260923 14:49:53.720474 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:53.720779 30731 external_mini_cluster.cc:1428] Setting key bec12b90b6b2f6e2c6c810c834737337
I20260923 14:49:53.725241 19850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7: Bootstrap starting.
I20260923 14:49:53.725950 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:53.722184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:59495 (local address 127.0.0.1:39225)
0923 14:49:53.722626 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:53.722640 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:53.722675 (+ 35us) server_negotiation.cc:407] Connection header received
0923 14:49:53.723297 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:53.723313 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:53.723594 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:53.723958 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:53.724734 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:53.724739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:53.724752 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:53.724763 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:53.724815 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:53.724835 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:53.724991 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:53.725065 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 14:49:53.725226 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":126,"threads_started":1}
I20260923 14:49:53.727514 19850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:53.728360 19850 log.cc:824] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:53.728662 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:53.730646 19850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7: No bootstrap required, opened a new log
I20260923 14:49:53.732455 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:53.730813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:56707 (local address 127.0.0.1:39225)
0923 14:49:53.730947 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:53.730951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:53.731134 (+ 183us) server_negotiation.cc:407] Connection header received
0923 14:49:53.731248 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:53.731251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:53.731277 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:53.731338 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:53.732055 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:53.732058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:53.732060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:53.732070 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:53.732093 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:53.732103 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:53.732181 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:53.732286 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 14:49:53.732316 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:49:53.734875 19850 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4b0f6eca6244bf8ac5e72a1c67bdd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260923 14:49:53.735081 19850 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:53.735114 19850 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4b0f6eca6244bf8ac5e72a1c67bdd7, State: Initialized, Role: FOLLOWER
I20260923 14:49:53.735468 19850 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4b0f6eca6244bf8ac5e72a1c67bdd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260923 14:49:53.735575 19850 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:53.735649 19850 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:53.735759 19850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:53.736794 19850 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4b0f6eca6244bf8ac5e72a1c67bdd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260923 14:49:53.737082 19850 leader_election.cc:302] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc4b0f6eca6244bf8ac5e72a1c67bdd7; no voters:
I20260923 14:49:53.737317 19850 leader_election.cc:288] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:53.737442 19860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:53.737689 19860 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [term 1 LEADER]: Becoming Leader. State: Replica: bc4b0f6eca6244bf8ac5e72a1c67bdd7, State: Running, Role: LEADER
I20260923 14:49:53.738010 19860 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4b0f6eca6244bf8ac5e72a1c67bdd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260923 14:49:53.738504 19850 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:53.739765 19862 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc4b0f6eca6244bf8ac5e72a1c67bdd7. Latest consensus state: current_term: 1 leader_uuid: "bc4b0f6eca6244bf8ac5e72a1c67bdd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4b0f6eca6244bf8ac5e72a1c67bdd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } } }
I20260923 14:49:53.739809 19861 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc4b0f6eca6244bf8ac5e72a1c67bdd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4b0f6eca6244bf8ac5e72a1c67bdd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } } }
I20260923 14:49:53.739934 19862 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:53.739939 19861 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:53.740391 19869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:53.743525 19869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:53.749346 19869 catalog_manager.cc:1347] Generated new cluster ID: e6692099e8da453897b52090e617a0ad
I20260923 14:49:53.749410 19869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:53.770807 19869 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:53.771154 19869 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:53.776286 19869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc4b0f6eca6244bf8ac5e72a1c67bdd7: Generated new TSK 0
I20260923 14:49:53.777379 19869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:53.789669 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 19782
2026-09-23T14:49:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:53 dist-test-slave-wjf2 krb5kdc[19881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:53 dist-test-slave-wjf2 krb5kdc[19881](info): set up 2 sockets
Sep 23 14:49:53 dist-test-slave-wjf2 krb5kdc[19881](info): commencing operation
krb5kdc: starting...
W20260923 14:49:54.837442 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:54 dist-test-slave-wjf2 krb5kdc[19881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:49:54.960052 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:42801
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:42801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:55.068486 19894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:55.068728 19894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:55.068768 19894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:55.072804 19894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:55.072857 19894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:55.072876 19894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:55.072892 19894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:55.072907 19894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:55.077986 19894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:42801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:42801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:55.078943 19894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:55.079959 19894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:55.088470 19900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:55.088490 19899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:55.088490 19902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:55.088759 19894 server_base.cc:1034] running on GCE node
I20260923 14:49:55.089303 19894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:55.090346 19894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:55.091512 19894 hybrid_clock.cc:648] HybridClock initialized: now 1790174995091475 us; error 48 us; skew 500 ppm
Sep 23 14:49:55 dist-test-slave-wjf2 krb5kdc[19881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174995, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:55.094274 19894 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:49:55.095165 19894 webserver.cc:460] Webserver started at http://127.30.2.254:37523/ using document root <none> and password file <none>
I20260923 14:49:55.095738 19894 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:55.095868 19894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:55.096097 19894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:55.098076 19894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "437ef86034e140b49983e90cbd4ad725"
format_stamp: "Formatted at 2026-09-23 14:49:55 on dist-test-slave-wjf2"
server_key: "b96644afe03869d99873b7953e81ea67"
server_key_iv: "a23e989686dda12883e8666128ef5f0f"
server_key_version: "encryptionkey@0"
I20260923 14:49:55.098646 19894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "437ef86034e140b49983e90cbd4ad725"
format_stamp: "Formatted at 2026-09-23 14:49:55 on dist-test-slave-wjf2"
server_key: "b96644afe03869d99873b7953e81ea67"
server_key_iv: "a23e989686dda12883e8666128ef5f0f"
server_key_version: "encryptionkey@0"
I20260923 14:49:55.102432 19894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260923 14:49:55.104928 19909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:55.106228 19894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:49:55.106365 19894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "437ef86034e140b49983e90cbd4ad725"
format_stamp: "Formatted at 2026-09-23 14:49:55 on dist-test-slave-wjf2"
server_key: "b96644afe03869d99873b7953e81ea67"
server_key_iv: "a23e989686dda12883e8666128ef5f0f"
server_key_version: "encryptionkey@0"
I20260923 14:49:55.106487 19894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:55.117161 19894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:55.119933 19894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:55.120127 19894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:55.127226 19894 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:42801
I20260923 14:49:55.127243 19961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:42801 every 8 connection(s)
I20260923 14:49:55.128137 19894 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:55.130975 19962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:55.135840 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 19894
I20260923 14:49:55.135999 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:55.136291 30731 external_mini_cluster.cc:1428] Setting key 934c6e85ca1243f3b2599dbf14abc04d
I20260923 14:49:55.137283 19962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725: Bootstrap starting.
I20260923 14:49:55.139883 19962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:55.140697 19962 log.cc:824] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:55.142735 19962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725: No bootstrap required, opened a new log
Sep 23 14:49:55 dist-test-slave-wjf2 krb5kdc[19881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:49:55.147344 19962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437ef86034e140b49983e90cbd4ad725" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42801 } }
I20260923 14:49:55.147574 19962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:55.147655 19962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437ef86034e140b49983e90cbd4ad725, State: Initialized, Role: FOLLOWER
I20260923 14:49:55.148222 19962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437ef86034e140b49983e90cbd4ad725" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42801 } }
I20260923 14:49:55.148360 19962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:55.148453 19962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:55.148581 19962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:55.149535 19962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437ef86034e140b49983e90cbd4ad725" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42801 } }
I20260923 14:49:55.149909 19962 leader_election.cc:302] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 437ef86034e140b49983e90cbd4ad725; no voters:
I20260923 14:49:55.150295 19962 leader_election.cc:288] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:55.150358 19967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:55.150557 19967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [term 1 LEADER]: Becoming Leader. State: Replica: 437ef86034e140b49983e90cbd4ad725, State: Running, Role: LEADER
I20260923 14:49:55.150928 19967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437ef86034e140b49983e90cbd4ad725" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42801 } }
I20260923 14:49:55.151254 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:55.137751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42461 (local address 127.30.2.254:42801)
0923 14:49:55.138210 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:55.138238 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:55.138274 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:49:55.138911 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:55.138927 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:55.139274 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:55.139617 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:55.140486 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:55.141524 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:55.142429 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:55.142954 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:55.143154 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:55.146022 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:55.146028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:55.146042 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:55.146068 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:55.148013 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:55.148572 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:55.148578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:55.148585 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:55.148745 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:55.149087 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:55.149090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:55.149092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:55.149486 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:55.149724 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:55.150008 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 14:49:55.150254 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":143,"threads_started":1}
I20260923 14:49:55.151666 19962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:55.152721 19969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "437ef86034e140b49983e90cbd4ad725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437ef86034e140b49983e90cbd4ad725" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42801 } } }
I20260923 14:49:55.152882 19969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:55.153321 19976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:55.153288 19968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 437ef86034e140b49983e90cbd4ad725. Latest consensus state: current_term: 1 leader_uuid: "437ef86034e140b49983e90cbd4ad725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437ef86034e140b49983e90cbd4ad725" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 42801 } } }
I20260923 14:49:55.153438 19968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:55.154909 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:55.156612 19976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:55.163275 19976 catalog_manager.cc:1347] Generated new cluster ID: 773fb78ebaa241c094414cb32f0b62da
I20260923 14:49:55.163337 19976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:55.166437 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:55.157460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49539 (local address 127.30.2.254:42801)
0923 14:49:55.157583 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:55.157588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:55.157906 (+ 318us) server_negotiation.cc:407] Connection header received
0923 14:49:55.158136 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:55.158140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:55.158193 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:55.158292 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:55.159071 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:55.159601 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:55.160460 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:55.160705 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:55.160824 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:55.162411 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:55.162417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:55.162421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:55.162456 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:55.164461 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:55.164995 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:55.165000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:55.165002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:55.165102 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:55.165452 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:55.165456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:55.165458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:55.165704 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:55.165869 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:55.166137 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 14:49:55.166251 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:49:55.188920 19976 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:55.190163 19976 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:55.194967 19976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 437ef86034e140b49983e90cbd4ad725: Generated new TSK 0
I20260923 14:49:55.195883 19976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:55.201284 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 19894
W20260923 14:49:55.207836 19879 connection.cc:535] client connection to 127.30.2.254:42801 recv error: Network error: failed to read from TLS socket (remote: 127.30.2.254:42801): Connection reset by peer (error 104)
2026-09-23T14:49:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:55 dist-test-slave-wjf2 krb5kdc[19993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:55 dist-test-slave-wjf2 krb5kdc[19993](info): set up 2 sockets
Sep 23 14:49:55 dist-test-slave-wjf2 krb5kdc[19993](info): commencing operation
krb5kdc: starting...
W20260923 14:49:56.273344 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:56 dist-test-slave-wjf2 krb5kdc[19993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:49:56.398984 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:38639
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:38639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:56.512480 20006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:56.512804 20006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:56.512887 20006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:56.516912 20006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:56.516997 20006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:56.517035 20006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:56.517091 20006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:56.517129 20006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:56.522450 20006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:38639
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:38639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:56.523682 20006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:56.524825 20006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:56.533565 20014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:56.533737 20006 server_base.cc:1034] running on GCE node
W20260923 14:49:56.533610 20011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:56.533540 20012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:56.534420 20006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:56.535488 20006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:56.536645 20006 hybrid_clock.cc:648] HybridClock initialized: now 1790174996536631 us; error 36 us; skew 500 ppm
Sep 23 14:49:56 dist-test-slave-wjf2 krb5kdc[19993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174996, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:56.540077 20006 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:49:56.540992 20006 webserver.cc:460] Webserver started at http://127.30.2.254:37501/ using document root <none> and password file <none>
I20260923 14:49:56.541543 20006 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:56.541603 20006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:56.541774 20006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:56.543915 20006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "95538793e2ff454fa14b2a023054d661"
format_stamp: "Formatted at 2026-09-23 14:49:56 on dist-test-slave-wjf2"
server_key: "765063609b72c2dc64300069d5de82a6"
server_key_iv: "2e3f6abd8dbd3c4449001fc8bcc84a1a"
server_key_version: "encryptionkey@0"
I20260923 14:49:56.544462 20006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "95538793e2ff454fa14b2a023054d661"
format_stamp: "Formatted at 2026-09-23 14:49:56 on dist-test-slave-wjf2"
server_key: "765063609b72c2dc64300069d5de82a6"
server_key_iv: "2e3f6abd8dbd3c4449001fc8bcc84a1a"
server_key_version: "encryptionkey@0"
I20260923 14:49:56.548338 20006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 14:49:56.550890 20021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:56.552244 20006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:49:56.552373 20006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "95538793e2ff454fa14b2a023054d661"
format_stamp: "Formatted at 2026-09-23 14:49:56 on dist-test-slave-wjf2"
server_key: "765063609b72c2dc64300069d5de82a6"
server_key_iv: "2e3f6abd8dbd3c4449001fc8bcc84a1a"
server_key_version: "encryptionkey@0"
I20260923 14:49:56.552474 20006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:56.570145 20006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:56.573758 20006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:56.573956 20006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:56.581799 20006 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:38639
I20260923 14:49:56.581820 20073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:38639 every 8 connection(s)
I20260923 14:49:56.582835 20006 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:56.585474 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 20006
I20260923 14:49:56.585613 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:56.585937 30731 external_mini_cluster.cc:1428] Setting key 5c7a494ab158e8f64e1a2a43fff4a88c
I20260923 14:49:56.586079 20074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:56.592595 20074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661: Bootstrap starting.
I20260923 14:49:56.595119 20074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:56.595444 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:56.587472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55619 (local address 127.30.2.254:38639)
0923 14:49:56.587988 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:56.588004 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:56.588053 (+ 49us) server_negotiation.cc:407] Connection header received
0923 14:49:56.588915 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:56.588934 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:56.589274 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:56.589612 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:56.590021 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:56.590996 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:56.591978 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:56.592452 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:56.592647 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:56.593600 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:56.593605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:56.593617 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:56.594036 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:56.594090 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:56.594113 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:56.594274 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:56.594464 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 14:49:56.594603 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":160,"threads_started":1}
I20260923 14:49:56.595981 20074 log.cc:824] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:56.598171 20074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661: No bootstrap required, opened a new log
I20260923 14:49:56.598452 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:56.603169 20074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95538793e2ff454fa14b2a023054d661" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 38639 } }
I20260923 14:49:56.603528 20074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:56.603600 20074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95538793e2ff454fa14b2a023054d661, State: Initialized, Role: FOLLOWER
I20260923 14:49:56.604147 20074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95538793e2ff454fa14b2a023054d661" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 38639 } }
I20260923 14:49:56.604372 20074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:56.604457 20074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:56.604565 20074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:56.605626 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:56.600754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33999 (local address 127.30.2.254:38639)
0923 14:49:56.600897 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:56.600902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:56.601017 (+ 115us) server_negotiation.cc:407] Connection header received
0923 14:49:56.601102 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:56.601105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:56.601149 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:56.601238 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:56.601485 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:56.602245 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:56.603410 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:56.603743 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:56.603912 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:56.605034 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:56.605040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:56.605043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:56.605059 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:56.605087 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:56.605101 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:56.605172 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:56.605441 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 14:49:56.605477 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:49:56.605702 20074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95538793e2ff454fa14b2a023054d661" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 38639 } }
I20260923 14:49:56.606087 20074 leader_election.cc:302] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95538793e2ff454fa14b2a023054d661; no voters:
I20260923 14:49:56.606479 20074 leader_election.cc:288] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:56.606531 20085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:56.606864 20085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [term 1 LEADER]: Becoming Leader. State: Replica: 95538793e2ff454fa14b2a023054d661, State: Running, Role: LEADER
I20260923 14:49:56.607246 20085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95538793e2ff454fa14b2a023054d661" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 38639 } }
I20260923 14:49:56.608112 20074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:56.609169 20086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95538793e2ff454fa14b2a023054d661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95538793e2ff454fa14b2a023054d661" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 38639 } } }
I20260923 14:49:56.609320 20086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:56.609700 20094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:56.609675 20087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95538793e2ff454fa14b2a023054d661. Latest consensus state: current_term: 1 leader_uuid: "95538793e2ff454fa14b2a023054d661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95538793e2ff454fa14b2a023054d661" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 38639 } } }
I20260923 14:49:56.609819 20087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661 [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:56.613454 20094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:56.619169 20094 catalog_manager.cc:1347] Generated new cluster ID: 49819e3ceedb44a88213554fe16b20b2
I20260923 14:49:56.619252 20094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:56.649034 20094 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:56.650465 20094 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:56.663533 20094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95538793e2ff454fa14b2a023054d661: Generated new TSK 0
I20260923 14:49:56.664392 20094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:56.679364 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 20006
W20260923 14:49:56.685537 19991 connection.cc:535] client connection to 127.30.2.254:38639 recv error: Network error: failed to read from TLS socket (remote: 127.30.2.254:38639): Connection reset by peer (error 104)
2026-09-23T14:49:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:56 dist-test-slave-wjf2 krb5kdc[20106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:56 dist-test-slave-wjf2 krb5kdc[20106](info): set up 2 sockets
Sep 23 14:49:56 dist-test-slave-wjf2 krb5kdc[20106](info): commencing operation
krb5kdc: starting...
W20260923 14:49:57.737360 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:57 dist-test-slave-wjf2 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:49:57.863058 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:37159
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:37159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:57.975023 20119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:57.975279 20119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:57.975320 20119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:57.979224 20119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:57.979275 20119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:57.979295 20119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:57.979310 20119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:57.979326 20119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:57.984529 20119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:37159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:37159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:57.985572 20119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:57.986567 20119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:57.995347 20125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:57.995478 20119 server_base.cc:1034] running on GCE node
W20260923 14:49:57.995375 20124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:57.995347 20127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:57.996250 20119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:57.997268 20119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:57.998458 20119 hybrid_clock.cc:648] HybridClock initialized: now 1790174997998441 us; error 28 us; skew 500 ppm
Sep 23 14:49:58 dist-test-slave-wjf2 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174998, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:58.004969 20119 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:49:58.005930 20119 webserver.cc:460] Webserver started at http://127.30.2.254:37559/ using document root <none> and password file <none>
I20260923 14:49:58.006553 20119 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:58.006649 20119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:58.006893 20119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:58.009032 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "65d0433c2da74f09aaada06b0e19a08a"
format_stamp: "Formatted at 2026-09-23 14:49:58 on dist-test-slave-wjf2"
server_key: "9ba2c14d4c4b7dd81bfbdc264f77d195"
server_key_iv: "f0d386a021be90b48ad01561deb368d4"
server_key_version: "encryptionkey@0"
I20260923 14:49:58.009609 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "65d0433c2da74f09aaada06b0e19a08a"
format_stamp: "Formatted at 2026-09-23 14:49:58 on dist-test-slave-wjf2"
server_key: "9ba2c14d4c4b7dd81bfbdc264f77d195"
server_key_iv: "f0d386a021be90b48ad01561deb368d4"
server_key_version: "encryptionkey@0"
I20260923 14:49:58.013411 20119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:49:58.015944 20135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:58.017277 20119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:49:58.017417 20119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "65d0433c2da74f09aaada06b0e19a08a"
format_stamp: "Formatted at 2026-09-23 14:49:58 on dist-test-slave-wjf2"
server_key: "9ba2c14d4c4b7dd81bfbdc264f77d195"
server_key_iv: "f0d386a021be90b48ad01561deb368d4"
server_key_version: "encryptionkey@0"
I20260923 14:49:58.017517 20119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:58.022096 20119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:58.025264 20119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:58.025475 20119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:58.032863 20119 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:37159
I20260923 14:49:58.032938 20194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:37159 every 8 connection(s)
I20260923 14:49:58.033888 20119 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:58.036731 20195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:58.039465 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 20119
I20260923 14:49:58.039623 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:58.040030 30731 external_mini_cluster.cc:1428] Setting key b188eb67666157f231d1f60c655dfbbf
I20260923 14:49:58.042658 20195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a: Bootstrap starting.
I20260923 14:49:58.045394 20195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:58.045776 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:58.041557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41485 (local address 127.30.2.254:37159)
0923 14:49:58.041994 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:58.042013 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:58.042055 (+ 42us) server_negotiation.cc:407] Connection header received
0923 14:49:58.042858 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:58.042884 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:58.043236 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:58.043588 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:58.044414 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:58.044420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:58.044435 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:58.044447 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:58.044509 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:58.044534 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:58.044666 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:58.044734 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 14:49:58.044880 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260923 14:49:58.046268 20195 log.cc:824] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:58.048421 20195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a: No bootstrap required, opened a new log
I20260923 14:49:58.048805 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:58.052726 20195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d0433c2da74f09aaada06b0e19a08a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37159 } }
I20260923 14:49:58.052981 20195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:58.053053 20195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d0433c2da74f09aaada06b0e19a08a, State: Initialized, Role: FOLLOWER
I20260923 14:49:58.053418 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:58.050961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39665 (local address 127.30.2.254:37159)
0923 14:49:58.051084 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:58.051088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:58.052053 (+ 965us) server_negotiation.cc:407] Connection header received
0923 14:49:58.052144 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:58.052147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:58.052180 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:58.052257 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:58.052944 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:58.052949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:58.052952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:49:58.052965 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:58.052994 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:49:58.053010 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:58.053103 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:58.053262 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 14:49:58.053299 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:49:58.053573 20195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d0433c2da74f09aaada06b0e19a08a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37159 } }
I20260923 14:49:58.053748 20195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:58.053820 20195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:58.053926 20195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:58.054877 20195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d0433c2da74f09aaada06b0e19a08a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37159 } }
I20260923 14:49:58.055166 20195 leader_election.cc:302] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65d0433c2da74f09aaada06b0e19a08a; no voters:
I20260923 14:49:58.055400 20195 leader_election.cc:288] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:58.055558 20205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:58.055960 20205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [term 1 LEADER]: Becoming Leader. State: Replica: 65d0433c2da74f09aaada06b0e19a08a, State: Running, Role: LEADER
I20260923 14:49:58.056427 20205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d0433c2da74f09aaada06b0e19a08a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37159 } }
I20260923 14:49:58.056692 20195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:58.058292 20206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65d0433c2da74f09aaada06b0e19a08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d0433c2da74f09aaada06b0e19a08a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37159 } } }
I20260923 14:49:58.058290 20207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65d0433c2da74f09aaada06b0e19a08a. Latest consensus state: current_term: 1 leader_uuid: "65d0433c2da74f09aaada06b0e19a08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d0433c2da74f09aaada06b0e19a08a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 37159 } } }
I20260923 14:49:58.058646 20207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:58.058586 20206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a [sys.catalog]: This master's current role is: LEADER
W20260923 14:49:58.061427 20219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:49:58.061514 20219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:49:58.061590 20220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:58.062263 20220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:58.069450 20220 catalog_manager.cc:1347] Generated new cluster ID: 46aec6fa73a149428e8b20b9ba55204f
I20260923 14:49:58.069518 20220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:58.095149 20220 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:58.095486 20220 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:58.107535 20220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65d0433c2da74f09aaada06b0e19a08a: Generated new TSK 0
I20260923 14:49:58.108613 20220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:58.124418 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 20119
2026-09-23T14:49:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:58 dist-test-slave-wjf2 krb5kdc[20226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:58 dist-test-slave-wjf2 krb5kdc[20226](info): set up 2 sockets
Sep 23 14:49:58 dist-test-slave-wjf2 krb5kdc[20226](info): commencing operation
krb5kdc: starting...
W20260923 14:49:59.161553 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:49:59 dist-test-slave-wjf2 krb5kdc[20226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:49:59.287477 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45961
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:45961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:49:59.406071 20239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:49:59.406363 20239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:49:59.406423 20239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:49:59.410878 20239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:49:59.410950 20239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:49:59.410972 20239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:49:59.410991 20239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:49:59.411007 20239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:49:59.420969 20239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:45961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:45961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:49:59.422173 20239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:49:59.423269 20239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:49:59.432217 20245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:59.432380 20239 server_base.cc:1034] running on GCE node
W20260923 14:49:59.432190 20247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:49:59.432211 20244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:49:59.433156 20239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:49:59.434376 20239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:49:59.435564 20239 hybrid_clock.cc:648] HybridClock initialized: now 1790174999435528 us; error 46 us; skew 500 ppm
Sep 23 14:49:59 dist-test-slave-wjf2 krb5kdc[20226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174999, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:49:59.441898 20239 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:49:59.442878 20239 webserver.cc:460] Webserver started at http://127.30.2.254:34345/ using document root <none> and password file <none>
I20260923 14:49:59.443552 20239 fs_manager.cc:359] Metadata directory not provided
I20260923 14:49:59.443686 20239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:49:59.443985 20239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:49:59.446316 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "46f0d102765a40a495ee168288f0a26f"
format_stamp: "Formatted at 2026-09-23 14:49:59 on dist-test-slave-wjf2"
server_key: "8557cba960b5df8a56763a6921667fa3"
server_key_iv: "6273b4ccc2886cc9ceb498f20d85e123"
server_key_version: "encryptionkey@0"
I20260923 14:49:59.446942 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "46f0d102765a40a495ee168288f0a26f"
format_stamp: "Formatted at 2026-09-23 14:49:59 on dist-test-slave-wjf2"
server_key: "8557cba960b5df8a56763a6921667fa3"
server_key_iv: "6273b4ccc2886cc9ceb498f20d85e123"
server_key_version: "encryptionkey@0"
I20260923 14:49:59.451231 20239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 14:49:59.454201 20254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:49:59.455907 20239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:49:59.456058 20239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "46f0d102765a40a495ee168288f0a26f"
format_stamp: "Formatted at 2026-09-23 14:49:59 on dist-test-slave-wjf2"
server_key: "8557cba960b5df8a56763a6921667fa3"
server_key_iv: "6273b4ccc2886cc9ceb498f20d85e123"
server_key_version: "encryptionkey@0"
I20260923 14:49:59.456187 20239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:49:59.480535 20239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:49:59.484164 20239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:49:59.484396 20239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:49:59.492044 20239 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:45961
I20260923 14:49:59.492065 20306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:45961 every 8 connection(s)
I20260923 14:49:59.493104 20239 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:49:59.494664 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 20239
I20260923 14:49:59.494777 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:49:59.495096 30731 external_mini_cluster.cc:1428] Setting key af7de1834a9ff5a07c5c10430b4c5589
I20260923 14:49:59.496639 20307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:49:59.503163 20307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f: Bootstrap starting.
Sep 23 14:49:59 dist-test-slave-wjf2 krb5kdc[20226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:49:59.506222 20307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f: Neither blocks nor log segments found. Creating new log.
I20260923 14:49:59.507077 20307 log.cc:824] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:49:59.509281 20307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f: No bootstrap required, opened a new log
I20260923 14:49:59.509969 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:59.496686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46057 (local address 127.30.2.254:45961)
0923 14:49:59.497161 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:59.497176 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:59.497215 (+ 39us) server_negotiation.cc:407] Connection header received
0923 14:49:59.497869 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:59.497884 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:59.498218 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:59.498549 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:59.499320 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:59.500431 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:59.501324 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:59.501897 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:59.502071 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:59.505076 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:59.505081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:59.505095 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:59.505127 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:59.507047 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:59.507560 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:59.507566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:59.507571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:59.507681 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:59.508015 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:59.508018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:59.508019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:59.508427 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:59.508637 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:59.508910 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 14:49:59.509199 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":158,"threads_started":1}
I20260923 14:49:59.513140 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:49:59.513973 20307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f0d102765a40a495ee168288f0a26f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45961 } }
I20260923 14:49:59.514231 20307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:49:59.514300 20307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f0d102765a40a495ee168288f0a26f, State: Initialized, Role: FOLLOWER
I20260923 14:49:59.514869 20307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f0d102765a40a495ee168288f0a26f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45961 } }
I20260923 14:49:59.515010 20307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:49:59.515139 20307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:49:59.515273 20307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:49:59.516702 20307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f0d102765a40a495ee168288f0a26f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45961 } }
I20260923 14:49:59.517091 20307 leader_election.cc:302] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46f0d102765a40a495ee168288f0a26f; no voters:
I20260923 14:49:59.517445 20307 leader_election.cc:288] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:49:59.517537 20317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:49:59.517827 20317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [term 1 LEADER]: Becoming Leader. State: Replica: 46f0d102765a40a495ee168288f0a26f, State: Running, Role: LEADER
I20260923 14:49:59.518244 20317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f0d102765a40a495ee168288f0a26f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45961 } }
I20260923 14:49:59.518862 20307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:49:59.519595 20319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46f0d102765a40a495ee168288f0a26f. Latest consensus state: current_term: 1 leader_uuid: "46f0d102765a40a495ee168288f0a26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f0d102765a40a495ee168288f0a26f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45961 } } }
I20260923 14:49:59.519834 20319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:59.520115 20318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46f0d102765a40a495ee168288f0a26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f0d102765a40a495ee168288f0a26f" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 45961 } } }
I20260923 14:49:59.520233 20318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f [sys.catalog]: This master's current role is: LEADER
I20260923 14:49:59.520452 20326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:49:59.524042 20326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:49:59.525141 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:49:59.515214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52981 (local address 127.30.2.254:45961)
0923 14:49:59.515330 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 14:49:59.515334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:49:59.515504 (+ 170us) server_negotiation.cc:407] Connection header received
0923 14:49:59.515650 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:49:59.515652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:49:59.515691 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:49:59.515762 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:49:59.516731 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:59.517359 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:59.518587 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:49:59.519026 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:49:59.519144 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:49:59.521081 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:49:59.521089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:49:59.521105 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:49:59.521132 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:49:59.523101 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:59.523607 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:59.523611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:59.523613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:59.523711 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:49:59.524046 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:49:59.524050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:49:59.524052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:49:59.524324 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:49:59.524460 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 14:49:59.524855 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 14:49:59.524982 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 14:49:59.530440 20326 catalog_manager.cc:1347] Generated new cluster ID: 159ca4ed879f488b863cdd72098f6a55
I20260923 14:49:59.530527 20326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:49:59.547415 20326 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:49:59.548523 20326 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:49:59.554944 20326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46f0d102765a40a495ee168288f0a26f: Generated new TSK 0
I20260923 14:49:59.555707 20326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:49:59.567930 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 20239
2026-09-23T14:49:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:49:59 dist-test-slave-wjf2 krb5kdc[20338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:49:59 dist-test-slave-wjf2 krb5kdc[20338](info): set up 2 sockets
Sep 23 14:49:59 dist-test-slave-wjf2 krb5kdc[20338](info): commencing operation
krb5kdc: starting...
W20260923 14:50:00.602324 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:50:00 dist-test-slave-wjf2 krb5kdc[20338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:50:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:50:00.730891 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44569
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:44569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:50:00.843982 20351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:50:00.844270 20351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:50:00.844347 20351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:50:00.848407 20351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:50:00.848474 20351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:50:00.848496 20351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:50:00.848558 20351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:50:00.848604 20351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:50:00.854045 20351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:44569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:44569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:50:00.855144 20351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:50:00.856245 20351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:50:00.865345 20357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:00.865414 20359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:00.865363 20356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:50:00.865572 20351 server_base.cc:1034] running on GCE node
I20260923 14:50:00.866379 20351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:50:00.867494 20351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:50:00.868664 20351 hybrid_clock.cc:648] HybridClock initialized: now 1790175000868633 us; error 43 us; skew 500 ppm
Sep 23 14:50:00 dist-test-slave-wjf2 krb5kdc[20338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175000, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:50:00.871526 20351 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:50:00.872493 20351 webserver.cc:460] Webserver started at http://127.30.2.254:39931/ using document root <none> and password file <none>
I20260923 14:50:00.873077 20351 fs_manager.cc:359] Metadata directory not provided
I20260923 14:50:00.873165 20351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:50:00.873401 20351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:50:00.875440 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "92d6f20bb194466199efbe97b885e43d"
format_stamp: "Formatted at 2026-09-23 14:50:00 on dist-test-slave-wjf2"
server_key: "dd8f38b31582a2493cd5f93702fe8dd7"
server_key_iv: "31a1d15de2e9a0b99e6a386cf543e3c0"
server_key_version: "encryptionkey@0"
I20260923 14:50:00.876161 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "92d6f20bb194466199efbe97b885e43d"
format_stamp: "Formatted at 2026-09-23 14:50:00 on dist-test-slave-wjf2"
server_key: "dd8f38b31582a2493cd5f93702fe8dd7"
server_key_iv: "31a1d15de2e9a0b99e6a386cf543e3c0"
server_key_version: "encryptionkey@0"
I20260923 14:50:00.880189 20351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260923 14:50:00.882929 20366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:50:00.884349 20351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:50:00.884495 20351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "92d6f20bb194466199efbe97b885e43d"
format_stamp: "Formatted at 2026-09-23 14:50:00 on dist-test-slave-wjf2"
server_key: "dd8f38b31582a2493cd5f93702fe8dd7"
server_key_iv: "31a1d15de2e9a0b99e6a386cf543e3c0"
server_key_version: "encryptionkey@0"
I20260923 14:50:00.884613 20351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:50:00.901558 20351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:50:00.905388 20351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:50:00.905611 20351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:50:00.913180 20351 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:44569
I20260923 14:50:00.913232 20418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:44569 every 8 connection(s)
I20260923 14:50:00.914251 20351 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:50:00.916798 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 20351
I20260923 14:50:00.916947 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:50:00.917161 20419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:50:00.917285 30731 external_mini_cluster.cc:1428] Setting key f7a512993fa8886316ffd31d28d4a7fd
I20260923 14:50:00.924649 20419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d: Bootstrap starting.
I20260923 14:50:00.927167 20419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d: Neither blocks nor log segments found. Creating new log.
I20260923 14:50:00.927657 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:00.918789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52723 (local address 127.30.2.254:44569)
0923 14:50:00.919209 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:00.919228 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:00.919272 (+ 44us) server_negotiation.cc:407] Connection header received
0923 14:50:00.920220 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:00.920245 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:00.920688 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:00.921076 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:00.921478 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:00.922943 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:00.923967 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:00.924620 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:00.924797 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:00.925675 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:00.925682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:00.925705 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:50:00.926205 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:00.926267 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:50:00.926296 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:00.926392 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:00.926507 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 14:50:00.926711 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":126,"threads_started":1}
I20260923 14:50:00.928056 20419 log.cc:824] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d: Log is configured to *not* fsync() on all Append() calls
I20260923 14:50:00.930192 20419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d: No bootstrap required, opened a new log
I20260923 14:50:00.930477 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:50:00.934367 20419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d6f20bb194466199efbe97b885e43d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44569 } }
I20260923 14:50:00.934638 20419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:50:00.934713 20419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d6f20bb194466199efbe97b885e43d, State: Initialized, Role: FOLLOWER
I20260923 14:50:00.935164 20419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d6f20bb194466199efbe97b885e43d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44569 } }
I20260923 14:50:00.935333 20419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:50:00.935410 20419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:50:00.935513 20419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:50:00.936578 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:00.932435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44157 (local address 127.30.2.254:44569)
0923 14:50:00.932545 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:00.932549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:00.932734 (+ 185us) server_negotiation.cc:407] Connection header received
0923 14:50:00.932887 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:00.932889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:00.932923 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:00.932985 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:00.933326 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:00.933829 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:00.934949 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:00.935230 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:00.935339 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:00.936177 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:00.936179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:00.936182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:50:00.936192 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:00.936229 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:50:00.936239 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:00.936297 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:00.936420 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 14:50:00.936459 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:50:00.936820 20419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d6f20bb194466199efbe97b885e43d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44569 } }
I20260923 14:50:00.937117 20419 leader_election.cc:302] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92d6f20bb194466199efbe97b885e43d; no voters:
I20260923 14:50:00.937430 20419 leader_election.cc:288] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:50:00.937522 20429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:50:00.937752 20429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [term 1 LEADER]: Becoming Leader. State: Replica: 92d6f20bb194466199efbe97b885e43d, State: Running, Role: LEADER
I20260923 14:50:00.938100 20429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d6f20bb194466199efbe97b885e43d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44569 } }
I20260923 14:50:00.938679 20419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:50:00.939713 20431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92d6f20bb194466199efbe97b885e43d. Latest consensus state: current_term: 1 leader_uuid: "92d6f20bb194466199efbe97b885e43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d6f20bb194466199efbe97b885e43d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44569 } } }
I20260923 14:50:00.939879 20431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [sys.catalog]: This master's current role is: LEADER
I20260923 14:50:00.940188 20430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92d6f20bb194466199efbe97b885e43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d6f20bb194466199efbe97b885e43d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 44569 } } }
I20260923 14:50:00.940266 20438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:50:00.940284 20430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d [sys.catalog]: This master's current role is: LEADER
I20260923 14:50:00.943197 20438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:50:00.948864 20438 catalog_manager.cc:1347] Generated new cluster ID: 30fa60ffd00048839d7d35de014fb614
I20260923 14:50:00.948927 20438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:50:00.972080 20438 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:50:00.973096 20438 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:50:00.976689 20438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92d6f20bb194466199efbe97b885e43d: Generated new TSK 0
I20260923 14:50:00.977415 20438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:50:00.982501 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 20351
2026-09-23T14:50:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:50:01 dist-test-slave-wjf2 krb5kdc[20450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:50:01 dist-test-slave-wjf2 krb5kdc[20450](info): set up 2 sockets
Sep 23 14:50:01 dist-test-slave-wjf2 krb5kdc[20450](info): commencing operation
krb5kdc: starting...
W20260923 14:50:02.097429 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:50:02 dist-test-slave-wjf2 krb5kdc[20450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:50:02.218703 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:46851
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:46851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:50:02.331211 20463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:50:02.331488 20463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:50:02.331532 20463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:50:02.335573 20463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:50:02.335638 20463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:50:02.335661 20463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:50:02.335682 20463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:50:02.335700 20463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:50:02.340894 20463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:46851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:46851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:50:02.341912 20463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:50:02.342926 20463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:50:02.351554 20469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:02.351572 20468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:02.351557 20471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:50:02.351886 20463 server_base.cc:1034] running on GCE node
I20260923 14:50:02.352491 20463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:50:02.353572 20463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:50:02.354763 20463 hybrid_clock.cc:648] HybridClock initialized: now 1790175002354729 us; error 42 us; skew 500 ppm
Sep 23 14:50:02 dist-test-slave-wjf2 krb5kdc[20450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175002, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:50:02.357609 20463 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:50:02.358549 20463 webserver.cc:460] Webserver started at http://127.30.2.254:33613/ using document root <none> and password file <none>
I20260923 14:50:02.359155 20463 fs_manager.cc:359] Metadata directory not provided
I20260923 14:50:02.359251 20463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:50:02.359484 20463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:50:02.361568 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "4e776b77374f40e28bca1ea4a85c10a0"
format_stamp: "Formatted at 2026-09-23 14:50:02 on dist-test-slave-wjf2"
server_key: "89e99dfe1c03feccc3ad5e03f80eda15"
server_key_iv: "c9107a3ca40ab6e72d15c6d42fd6a179"
server_key_version: "encryptionkey@0"
I20260923 14:50:02.362170 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "4e776b77374f40e28bca1ea4a85c10a0"
format_stamp: "Formatted at 2026-09-23 14:50:02 on dist-test-slave-wjf2"
server_key: "89e99dfe1c03feccc3ad5e03f80eda15"
server_key_iv: "c9107a3ca40ab6e72d15c6d42fd6a179"
server_key_version: "encryptionkey@0"
I20260923 14:50:02.366199 20463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:50:02.368656 20478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:50:02.370033 20463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:50:02.370178 20463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "4e776b77374f40e28bca1ea4a85c10a0"
format_stamp: "Formatted at 2026-09-23 14:50:02 on dist-test-slave-wjf2"
server_key: "89e99dfe1c03feccc3ad5e03f80eda15"
server_key_iv: "c9107a3ca40ab6e72d15c6d42fd6a179"
server_key_version: "encryptionkey@0"
I20260923 14:50:02.370287 20463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:50:02.374858 20463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:50:02.378103 20463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:50:02.378295 20463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:50:02.385790 20463 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:46851
I20260923 14:50:02.385836 20530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:46851 every 8 connection(s)
I20260923 14:50:02.386783 20463 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:50:02.389809 20531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:50:02.394738 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 20463
I20260923 14:50:02.394877 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:50:02.395136 30731 external_mini_cluster.cc:1428] Setting key a3c3b7d43629d4e6e9877429d224f03f
I20260923 14:50:02.395967 20531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0: Bootstrap starting.
I20260923 14:50:02.398794 20531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0: Neither blocks nor log segments found. Creating new log.
I20260923 14:50:02.399811 20531 log.cc:824] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0: Log is configured to *not* fsync() on all Append() calls
I20260923 14:50:02.401154 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:02.396643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35273 (local address 127.30.2.254:46851)
0923 14:50:02.397118 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:02.397145 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:02.397210 (+ 65us) server_negotiation.cc:407] Connection header received
0923 14:50:02.398039 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:02.398064 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:02.398452 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:02.398831 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:02.399560 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:02.399566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:02.399582 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:50:02.399596 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:02.399661 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:50:02.399696 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:02.399846 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:02.400195 (+ 349us) server_negotiation.cc:299] Negotiation successful
0923 14:50:02.400390 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":109,"threads_started":1}
I20260923 14:50:02.401961 20531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0: No bootstrap required, opened a new log
I20260923 14:50:02.404299 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:50:02.406738 20531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e776b77374f40e28bca1ea4a85c10a0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46851 } }
I20260923 14:50:02.407064 20531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:50:02.407147 20531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e776b77374f40e28bca1ea4a85c10a0, State: Initialized, Role: FOLLOWER
I20260923 14:50:02.407683 20531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e776b77374f40e28bca1ea4a85c10a0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46851 } }
I20260923 14:50:02.407975 20531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:50:02.408098 20531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:50:02.408221 20531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:50:02.408828 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:02.406738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42709 (local address 127.30.2.254:46851)
0923 14:50:02.406884 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:02.406889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:02.407135 (+ 246us) server_negotiation.cc:407] Connection header received
0923 14:50:02.407350 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:02.407352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:02.407399 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:02.407488 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:02.408365 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:02.408369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:02.408371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:50:02.408383 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:02.408416 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:50:02.408427 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:02.408500 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:02.408646 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 14:50:02.408677 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:50:02.409487 20531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e776b77374f40e28bca1ea4a85c10a0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46851 } }
I20260923 14:50:02.409866 20531 leader_election.cc:302] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e776b77374f40e28bca1ea4a85c10a0; no voters:
I20260923 14:50:02.410270 20531 leader_election.cc:288] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:50:02.410347 20541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:50:02.410605 20541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e776b77374f40e28bca1ea4a85c10a0, State: Running, Role: LEADER
I20260923 14:50:02.410918 20541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e776b77374f40e28bca1ea4a85c10a0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46851 } }
I20260923 14:50:02.411834 20531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:50:02.413225 20543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e776b77374f40e28bca1ea4a85c10a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e776b77374f40e28bca1ea4a85c10a0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46851 } } }
I20260923 14:50:02.413362 20543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:50:02.413673 20542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e776b77374f40e28bca1ea4a85c10a0. Latest consensus state: current_term: 1 leader_uuid: "4e776b77374f40e28bca1ea4a85c10a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e776b77374f40e28bca1ea4a85c10a0" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46851 } } }
I20260923 14:50:02.413810 20542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:50:02.414856 20550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:50:02.417621 20550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:50:02.422871 20550 catalog_manager.cc:1347] Generated new cluster ID: 4ba171e819a141a5960e1cf0744ebfaf
I20260923 14:50:02.422945 20550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:50:02.437353 20550 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:50:02.437696 20550 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:50:02.450585 20550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e776b77374f40e28bca1ea4a85c10a0: Generated new TSK 0
I20260923 14:50:02.451344 20550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:50:02.462440 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 20463
2026-09-23T14:50:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1479 ms)
[----------] 18 tests from AuthTokenIssuingTest (25784 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:50:02 dist-test-slave-wjf2 krb5kdc[20562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:50:02 dist-test-slave-wjf2 krb5kdc[20562](info): set up 2 sockets
Sep 23 14:50:02 dist-test-slave-wjf2 krb5kdc[20562](info): commencing operation
krb5kdc: starting...
W20260923 14:50:03.513619 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:50:03 dist-test-slave-wjf2 krb5kdc[20562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:50:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:50:03.640022 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:46277
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:46277,127.30.2.253:43531,127.30.2.252:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:50:03.749814 20575 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:50:03.750133 20575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:50:03.750198 20575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:50:03.750270 20575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:50:03.754288 20575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:50:03.754364 20575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:50:03.754403 20575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:50:03.754454 20575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:50:03.754495 20575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:50:03.759647 20575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:46277,127.30.2.253:43531,127.30.2.252:42177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:46277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:50:03.760701 20575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:50:03.761773 20575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:50:03.770484 20583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:03.770484 20581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:50:03.770495 20575 server_base.cc:1034] running on GCE node
W20260923 14:50:03.770476 20580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:50:03.771266 20575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:50:03.772397 20575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:50:03.773579 20575 hybrid_clock.cc:648] HybridClock initialized: now 1790175003773559 us; error 31 us; skew 500 ppm
Sep 23 14:50:03 dist-test-slave-wjf2 krb5kdc[20562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:50:03.776346 20575 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:50:03.777274 20575 webserver.cc:460] Webserver started at http://127.30.2.254:45805/ using document root <none> and password file <none>
I20260923 14:50:03.777858 20575 fs_manager.cc:359] Metadata directory not provided
I20260923 14:50:03.777943 20575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:50:03.778167 20575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:50:03.780288 20575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "9d62b4b2edba41d89b520475c4529f7a"
format_stamp: "Formatted at 2026-09-23 14:50:03 on dist-test-slave-wjf2"
server_key: "bd93de3214354931d6c9cc388f31c95b"
server_key_iv: "b516d60a254804a4dc6bed45abe81125"
server_key_version: "encryptionkey@0"
I20260923 14:50:03.780860 20575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "9d62b4b2edba41d89b520475c4529f7a"
format_stamp: "Formatted at 2026-09-23 14:50:03 on dist-test-slave-wjf2"
server_key: "bd93de3214354931d6c9cc388f31c95b"
server_key_iv: "b516d60a254804a4dc6bed45abe81125"
server_key_version: "encryptionkey@0"
I20260923 14:50:03.784668 20575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:50:03.787173 20590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:50:03.788455 20575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:50:03.788589 20575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "9d62b4b2edba41d89b520475c4529f7a"
format_stamp: "Formatted at 2026-09-23 14:50:03 on dist-test-slave-wjf2"
server_key: "bd93de3214354931d6c9cc388f31c95b"
server_key_iv: "b516d60a254804a4dc6bed45abe81125"
server_key_version: "encryptionkey@0"
I20260923 14:50:03.788699 20575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:50:03.807550 20575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:50:03.810925 20575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:50:03.811200 20575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:50:03.818348 20575 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:46277
I20260923 14:50:03.818727 20643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:46277 every 8 connection(s)
I20260923 14:50:03.819373 20575 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:50:03.822402 20644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:50:03.826170 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 20575
I20260923 14:50:03.826326 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:50:03.826594 30731 external_mini_cluster.cc:1428] Setting key 97b9f4183e1f631bfce3e612a51be371
I20260923 14:50:03.826769 20644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.2.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.253@KRBTEST.COM" created.
Sep 23 14:50:03 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:50:03.845659 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:03.829364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:46277 (local address 127.0.0.1:49382)
0923 14:50:03.829930 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:03.829993 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:03.831474 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:03.832350 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:03.832360 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:03.832954 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:03.833407 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:03.833433 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:03.834578 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:03.834585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:03.836107 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:03.836123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:03.836824 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:03.836830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:03.836985 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:03.837489 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:03.837659 (+ 170us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:03.840362 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:03.842604 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:03.842613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:03.842622 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:03.842852 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:03.843280 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:03.843282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:03.843283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:03.843388 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:03.844226 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:03.844233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:03.844386 (+ 153us) client_negotiation.cc:769] Sending connection context
0923 14:50:03.844532 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 14:50:03.844808 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":45,"thread_start_us":127,"threads_started":1}
I20260923 14:50:03.845665 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:03.829364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.30.2.254:46277)
0923 14:50:03.829855 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:03.829873 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:03.830124 (+ 251us) server_negotiation.cc:407] Connection header received
0923 14:50:03.831918 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:03.831923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:03.832117 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:03.832272 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:03.833586 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:03.834439 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:03.836237 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:03.836645 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:03.836793 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:03.840517 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:03.840523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:03.840534 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:03.840556 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:03.842505 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:03.843004 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:03.843014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:03.843019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:03.843167 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:03.843481 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:03.843487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:03.843492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:03.844090 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:03.844248 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:03.844651 (+ 403us) server_negotiation.cc:299] Negotiation successful
0923 14:50:03.844915 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":352,"spinlock_wait_cycles":39168,"thread_start_us":180,"threads_started":1}
I20260923 14:50:03.848454 20644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:03.849177 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:03.848931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43531 (local address 127.0.0.1:39144)
0923 14:50:03.849050 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:03.849120 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:43531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 14:50:03.849386 20594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.2.253:43531: connect: Connection refused (error 111)
I20260923 14:50:03.850198 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:03.850016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43531 (local address 127.0.0.1:39146)
0923 14:50:03.850108 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:03.850149 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:43531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 14:50:03.850466 20644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.253:43531: Network error: Client connection negotiation failed: client connection to 127.30.2.253:43531: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab.
Entry for principal kudu/127.30.2.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab.
I20260923 14:50:03.857167 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.253:43531
--webserver_interface=127.30.2.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab
--principal=kudu/127.30.2.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:46277,127.30.2.253:43531,127.30.2.252:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:50:03.899753 20644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } attempt: 1
I20260923 14:50:03.900768 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:03.900413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43531 (local address 127.0.0.1:39154)
0923 14:50:03.900597 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:03.900676 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:43531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 14:50:03.901721 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:03.901462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43531 (local address 127.0.0.1:39166)
0923 14:50:03.901577 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:03.901641 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:43531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 14:50:03.901878 20644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.253:43531: Network error: Client connection negotiation failed: client connection to 127.30.2.253:43531: connect: Connection refused (error 111)
I20260923 14:50:03.970041 20644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } attempt: 2
I20260923 14:50:03.970907 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:03.970577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43531 (local address 127.0.0.1:39180)
0923 14:50:03.970742 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:03.970819 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:43531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 14:50:03.970705 20651 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:50:03.971061 20651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:50:03.971132 20651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:50:03.971227 20651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 14:50:03.971668 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:03.971430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43531 (local address 127.0.0.1:39192)
0923 14:50:03.971552 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:03.971603 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:43531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 14:50:03.971822 20644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.253:43531: Network error: Client connection negotiation failed: client connection to 127.30.2.253:43531: connect: Connection refused (error 111)
W20260923 14:50:03.975406 20651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:50:03.975457 20651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:50:03.975476 20651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:50:03.975493 20651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:50:03.975508 20651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:50:03.980719 20651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:46277,127.30.2.253:43531,127.30.2.252:42177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab
--principal=kudu/127.30.2.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.253:43531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.2.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:50:03.981688 20651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:50:03.982679 20651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:50:03.991323 20659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:03.991367 20657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:03.991323 20656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:50:03.991528 20651 server_base.cc:1034] running on GCE node
I20260923 14:50:03.992547 20651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:50:03.993700 20651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:50:03.994915 20651 hybrid_clock.cc:648] HybridClock initialized: now 1790175003994897 us; error 30 us; skew 500 ppm
Sep 23 14:50:03 dist-test-slave-wjf2 krb5kdc[20562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:50:03.997690 20651 init.cc:377] Logged in from keytab as kudu/127.30.2.253@KRBTEST.COM (short username kudu)
I20260923 14:50:03.998611 20651 webserver.cc:460] Webserver started at http://127.30.2.253:39453/ using document root <none> and password file <none>
I20260923 14:50:03.999181 20651 fs_manager.cc:359] Metadata directory not provided
I20260923 14:50:03.999265 20651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:50:03.999496 20651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:50:04.001614 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/data/instance:
uuid: "ac38f8d4e1194a12a5ee39b3e73caa53"
format_stamp: "Formatted at 2026-09-23 14:50:03 on dist-test-slave-wjf2"
server_key: "8818f4d51c097befbda7e92f4a45fd3e"
server_key_iv: "06e82282521be0cb64fdb36704789bda"
server_key_version: "encryptionkey@0"
I20260923 14:50:04.002182 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/wal/instance:
uuid: "ac38f8d4e1194a12a5ee39b3e73caa53"
format_stamp: "Formatted at 2026-09-23 14:50:03 on dist-test-slave-wjf2"
server_key: "8818f4d51c097befbda7e92f4a45fd3e"
server_key_iv: "06e82282521be0cb64fdb36704789bda"
server_key_version: "encryptionkey@0"
I20260923 14:50:04.005962 20651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:50:04.008462 20666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:50:04.009804 20651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:50:04.009955 20651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/wal
uuid: "ac38f8d4e1194a12a5ee39b3e73caa53"
format_stamp: "Formatted at 2026-09-23 14:50:03 on dist-test-slave-wjf2"
server_key: "8818f4d51c097befbda7e92f4a45fd3e"
server_key_iv: "06e82282521be0cb64fdb36704789bda"
server_key_version: "encryptionkey@0"
I20260923 14:50:04.010067 20651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:50:04.038591 20651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:50:04.041529 20651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:50:04.041731 20651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:50:04.048879 20651 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.253:43531
I20260923 14:50:04.049183 20723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.253:43531 every 8 connection(s)
I20260923 14:50:04.049880 20651 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
I20260923 14:50:04.052762 20724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:50:04.053653 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 20651
I20260923 14:50:04.053789 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-1/wal/instance
I20260923 14:50:04.054121 30731 external_mini_cluster.cc:1428] Setting key a232deff362351c5978dc305606fd714
I20260923 14:50:04.057519 20724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.2.252@KRBTEST.COM; defaulting to no policy
I20260923 14:50:04.062970 20644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.252@KRBTEST.COM" created.
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
I20260923 14:50:04.073648 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.059492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.30.2.254:46277)
0923 14:50:04.059680 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.059686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.060533 (+ 847us) server_negotiation.cc:407] Connection header received
0923 14:50:04.061478 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.061482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.061550 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.061646 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.062908 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.063679 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.065470 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.065732 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.065844 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.068624 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.068628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.068630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.068652 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.070986 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.071534 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.071538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.071540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.071632 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.072175 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.072181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.072183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.072435 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.072540 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.073303 (+ 763us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.073444 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 14:50:04.074630 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.063435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43531 (local address 127.0.0.1:39200)
0923 14:50:04.063559 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.063574 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:04.063649 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:04.064464 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:04.064466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:04.064714 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:04.064868 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.064877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.066680 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.066684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.067530 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.067540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.068168 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.068172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.068279 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.068765 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:04.068777 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:04.070314 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:04.073028 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.073031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.073033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.073222 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.073532 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.073535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.073537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.073622 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.074124 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:04.074128 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:04.074222 (+ 94us) client_negotiation.cc:769] Sending connection context
0923 14:50:04.074339 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 14:50:04.074474 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 14:50:04.075446 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.059889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:46277 (local address 127.0.0.1:49390)
0923 14:50:04.060364 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.060416 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:04.061266 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:04.061834 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:04.061841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:04.062256 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:04.062747 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.062761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.063865 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.063869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.065347 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.065358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.065889 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.065893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.066067 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.066709 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:04.066730 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:04.068483 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:04.071076 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.071087 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.071098 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.071361 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.071754 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.071760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.071764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.072016 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.072564 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:04.072572 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:04.072941 (+ 369us) client_negotiation.cc:769] Sending connection context
0923 14:50:04.074038 (+ 1097us) client_negotiation.cc:240] Negotiation successful
0923 14:50:04.074439 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":42,"thread_start_us":133,"threads_started":1}
I20260923 14:50:04.075449 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.063552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.30.2.253:43531)
0923 14:50:04.063986 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.063998 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.064020 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:50:04.064060 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.064066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.064265 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.064441 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.064998 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.066199 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.067647 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.068049 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.068162 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.070480 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.070491 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.070506 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.070535 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.072908 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.073344 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.073350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.073352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.073436 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.073721 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.073725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.073728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.074012 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.074145 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.074426 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.074568 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":333,"thread_start_us":237,"threads_started":1}
I20260923 14:50:04.076794 20724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:04.077661 20644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:04.078352 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.078108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46548)
0923 14:50:04.078225 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.078287 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 14:50:04.078990 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.078734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46552)
0923 14:50:04.078889 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.078941 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260923 14:50:04.079135 20644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:42177: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab.
Entry for principal kudu/127.30.2.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab.
I20260923 14:50:04.083539 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.252:42177
--webserver_interface=127.30.2.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab
--principal=kudu/127.30.2.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:46277,127.30.2.253:43531,127.30.2.252:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:50:04.085741 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.077361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.30.2.253:43531)
0923 14:50:04.077465 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.077469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.077518 (+ 49us) server_negotiation.cc:407] Connection header received
0923 14:50:04.077653 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.077655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.077700 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.077788 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.078258 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.078996 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.079961 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.080204 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.080311 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.082281 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.082285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.082286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.082307 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.084020 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.084425 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.084431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.084436 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.084529 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.084778 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.084785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.084789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.085033 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.085146 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.085431 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.085588 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":31}
I20260923 14:50:04.085774 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.077324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43531 (local address 127.0.0.1:39216)
0923 14:50:04.077465 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.077478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:04.077561 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:04.077818 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:04.077822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:04.077997 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:04.078116 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.078124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.079102 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.079105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.079823 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.079834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.080308 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.080311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.080428 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.080886 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:04.080897 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:04.082112 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:04.084128 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.084131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.084132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.084330 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.084625 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.084627 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.084628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.084698 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.085148 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:04.085152 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:04.085261 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 14:50:04.085382 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 14:50:04.085538 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":18}
I20260923 14:50:04.086658 20724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:04.087311 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.087076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46566)
0923 14:50:04.087179 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.087256 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 14:50:04.088040 20667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
I20260923 14:50:04.088989 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.088803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46576)
0923 14:50:04.088904 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.088945 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 14:50:04.089205 20724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:42177: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
I20260923 14:50:04.110306 20644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } attempt: 1
I20260923 14:50:04.111120 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.110827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46584)
0923 14:50:04.110969 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.111037 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 14:50:04.111941 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.111644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46586)
0923 14:50:04.111833 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.111888 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 14:50:04.112069 20644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:42177: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
I20260923 14:50:04.138495 20724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } attempt: 1
I20260923 14:50:04.139441 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.139124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46602)
0923 14:50:04.139292 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.139358 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 14:50:04.140213 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.139997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46606)
0923 14:50:04.140105 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.140154 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 14:50:04.140341 20724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:42177: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
I20260923 14:50:04.187274 20644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } attempt: 2
I20260923 14:50:04.188211 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.187909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46608)
0923 14:50:04.188064 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.188128 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 14:50:04.188891 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.188681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46622)
0923 14:50:04.188788 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.188837 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 14:50:04.189018 20644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:42177: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
W20260923 14:50:04.198117 20730 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:50:04.198414 20730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:50:04.198472 20730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:50:04.198558 20730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:50:04.202556 20730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:50:04.202625 20730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:50:04.202672 20730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:50:04.202724 20730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:50:04.202761 20730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:50:04.207959 20730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:46277,127.30.2.253:43531,127.30.2.252:42177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab
--principal=kudu/127.30.2.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.252:42177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.2.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:50:04.208491 20724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } attempt: 2
I20260923 14:50:04.209059 20730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:50:04.209277 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.208978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46626)
0923 14:50:04.209117 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.209182 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 14:50:04.210057 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.209851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46628)
0923 14:50:04.209956 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.210000 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 14:50:04.210175 20724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:42177: Network error: Client connection negotiation failed: client connection to 127.30.2.252:42177: connect: Connection refused (error 111)
I20260923 14:50:04.210307 20730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:50:04.218955 20736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:04.218921 20737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:04.218921 20739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:50:04.219030 20730 server_base.cc:1034] running on GCE node
I20260923 14:50:04.219692 20730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:50:04.220817 20730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:50:04.221973 20730 hybrid_clock.cc:648] HybridClock initialized: now 1790175004221956 us; error 30 us; skew 500 ppm
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175004, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:50:04.224822 20730 init.cc:377] Logged in from keytab as kudu/127.30.2.252@KRBTEST.COM (short username kudu)
I20260923 14:50:04.225696 20730 webserver.cc:460] Webserver started at http://127.30.2.252:41125/ using document root <none> and password file <none>
I20260923 14:50:04.226264 20730 fs_manager.cc:359] Metadata directory not provided
I20260923 14:50:04.226323 20730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:50:04.226508 20730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:50:04.228621 20730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/data/instance:
uuid: "b954a44e9bdf4d9599302d09fa0e3bb6"
format_stamp: "Formatted at 2026-09-23 14:50:04 on dist-test-slave-wjf2"
server_key: "71cf159e7a087953aca4c6878327b060"
server_key_iv: "77faaabc4548948f41a06744d61786b8"
server_key_version: "encryptionkey@0"
I20260923 14:50:04.229157 20730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/wal/instance:
uuid: "b954a44e9bdf4d9599302d09fa0e3bb6"
format_stamp: "Formatted at 2026-09-23 14:50:04 on dist-test-slave-wjf2"
server_key: "71cf159e7a087953aca4c6878327b060"
server_key_iv: "77faaabc4548948f41a06744d61786b8"
server_key_version: "encryptionkey@0"
I20260923 14:50:04.233187 20730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 14:50:04.235670 20746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:50:04.237042 20730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:50:04.237159 20730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/wal
uuid: "b954a44e9bdf4d9599302d09fa0e3bb6"
format_stamp: "Formatted at 2026-09-23 14:50:04 on dist-test-slave-wjf2"
server_key: "71cf159e7a087953aca4c6878327b060"
server_key_iv: "77faaabc4548948f41a06744d61786b8"
server_key_version: "encryptionkey@0"
I20260923 14:50:04.237251 20730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:50:04.276772 20730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:50:04.279953 20730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:50:04.280117 20730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:50:04.287364 20730 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.252:42177
I20260923 14:50:04.287644 20799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.252:42177 every 8 connection(s)
I20260923 14:50:04.288177 20644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } attempt: 3
I20260923 14:50:04.288520 20730 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
I20260923 14:50:04.291141 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 20730
I20260923 14:50:04.291268 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174723338690-30731-0/minicluster-data/master-2/wal/instance
I20260923 14:50:04.291612 30731 external_mini_cluster.cc:1428] Setting key 5be53fb450225379868eecada90d9a4a
I20260923 14:50:04.291815 20800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
I20260923 14:50:04.297375 20800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
I20260923 14:50:04.301602 20724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } attempt: 3
I20260923 14:50:04.302197 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.288741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46630)
0923 14:50:04.288944 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.288962 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:04.289076 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:04.291609 (+ 2533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:04.291613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:04.291848 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:04.291979 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.291988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.293723 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.293728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.294579 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.294592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.295561 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.295567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.295660 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.296204 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:04.296229 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:04.297687 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:04.300178 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.300181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.300184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.300424 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.300865 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.300869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.300871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.300971 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.301713 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:04.301716 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:04.301807 (+ 91us) client_negotiation.cc:769] Sending connection context
0923 14:50:04.301893 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 14:50:04.302028 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 14:50:04.304010 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.289579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.30.2.252:42177)
0923 14:50:04.290012 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.290027 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.290063 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:50:04.290748 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.290765 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.291247 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.291660 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.292151 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.293586 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.294711 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.295420 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.295587 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.297971 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.297977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.297996 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.298028 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.300068 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.300534 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.300540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.300546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.300721 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.301076 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.301079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.301081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.301586 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.302561 (+ 975us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.302717 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.302985 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":138,"threads_started":1}
I20260923 14:50:04.304834 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.292275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.30.2.252:42177)
0923 14:50:04.292703 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.292712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.292734 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:50:04.292896 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.292899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.292941 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.293028 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.293970 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.294779 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.295633 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.295984 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.296098 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.300325 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.300330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.300333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.300357 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.302639 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.303148 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.303152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.303154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.303238 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.303547 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.303552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.303554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.303768 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.303960 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.304449 (+ 489us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.304581 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":362,"thread_start_us":129,"threads_started":1}
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175004, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
I20260923 14:50:04.309700 20644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a: Bootstrap starting.
I20260923 14:50:04.310740 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.299317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:46277 (local address 127.0.0.1:49406)
0923 14:50:04.299710 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.299763 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:04.299955 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:04.300374 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:04.300381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:04.300800 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:04.300949 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.300965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.301916 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.301919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.302992 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.303002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.303689 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.303692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.303866 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.304973 (+ 1107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:04.304987 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:04.306566 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:04.308860 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.308872 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.308880 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.309155 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.309472 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.309475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.309477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.309563 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.310043 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:04.310051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:04.310178 (+ 127us) client_negotiation.cc:769] Sending connection context
0923 14:50:04.310340 (+ 162us) client_negotiation.cc:240] Negotiation successful
0923 14:50:04.310570 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":21,"thread_start_us":130,"threads_started":1}
I20260923 14:50:04.310863 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.299376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.30.2.254:46277)
0923 14:50:04.299517 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.299522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.299860 (+ 338us) server_negotiation.cc:407] Connection header received
0923 14:50:04.300072 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.300076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.300127 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.300180 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.301080 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.301794 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.303138 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.303454 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.303528 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.306730 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.306736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.306741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.306774 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.308723 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.309256 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.309275 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.309279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.309365 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.309665 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.309668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.309670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.309901 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.309988 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.310614 (+ 626us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.310718 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:50:04.312206 20800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:04.313151 20644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a: Neither blocks nor log segments found. Creating new log.
I20260923 14:50:04.314219 20644 log.cc:824] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:50:04.314456 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.302047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46640)
0923 14:50:04.302353 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.302366 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:04.302444 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:04.303123 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:04.303128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:04.303341 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:04.303457 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.303468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.305811 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.305818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.306660 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.306673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.307290 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.307294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.307398 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.307967 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:04.307981 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:04.309481 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:04.312783 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.312787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.312789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.313085 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.313414 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.313420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.313424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.313521 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.313966 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:04.313969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:04.314077 (+ 108us) client_negotiation.cc:769] Sending connection context
0923 14:50:04.314158 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 14:50:04.314294 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260923 14:50:04.315122 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.302588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.30.2.252:42177)
0923 14:50:04.302863 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.302873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.302885 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:50:04.302916 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.302918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.302962 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.303565 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.303678 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.305682 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.306800 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.307153 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.307597 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.309634 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.309637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.309640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.309663 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.312647 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.313223 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.313229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.313231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.313317 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.313627 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.313630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.313632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.313842 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.314728 (+ 886us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.314807 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.314928 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
I20260923 14:50:04.316529 20644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a: No bootstrap required, opened a new log
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175004, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
I20260923 14:50:04.318295 20724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53: Bootstrap starting.
I20260923 14:50:04.321182 20724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53: Neither blocks nor log segments found. Creating new log.
I20260923 14:50:04.321323 20644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } }
I20260923 14:50:04.321557 20644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d62b4b2edba41d89b520475c4529f7a, State: Initialized, Role: FOLLOWER
I20260923 14:50:04.321825 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.306936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.30.2.254:46277)
0923 14:50:04.307611 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.307624 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.307637 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:50:04.307681 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.307684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.307728 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.307859 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.309245 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.310029 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.311087 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.311843 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.312347 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.317652 (+ 5305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.317655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.317658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.317682 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.319382 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.319931 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.319935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.319938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.320034 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.320321 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.320324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.320326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.320525 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.320730 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.321428 (+ 698us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.321547 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260923 14:50:04.321936 20644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } }
I20260923 14:50:04.322032 20724 log.cc:824] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53: Log is configured to *not* fsync() on all Append() calls
I20260923 14:50:04.322348 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.312665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:43531 (local address 127.0.0.1:39222)
0923 14:50:04.312864 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.312886 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:04.312958 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:04.313362 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:04.313364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:04.313545 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:04.313648 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.313654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.314691 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.314698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.315597 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.315608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.316235 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.316239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.316321 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.316785 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:04.316798 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:04.318371 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:04.320638 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.320641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.320643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.320937 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.321242 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.321245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.321247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.321332 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.321851 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:04.321854 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:04.321943 (+ 89us) client_negotiation.cc:769] Sending connection context
0923 14:50:04.322047 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 14:50:04.322179 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 14:50:04.322779 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.312928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.30.2.253:43531)
0923 14:50:04.313048 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.313052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.313063 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:50:04.313136 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.313139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.313180 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.313249 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.313760 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.314594 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.315723 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.316106 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.316225 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.318781 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.318785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.318788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.318815 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.320502 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.321057 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.321060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.321062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.321143 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.321445 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.321449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.321450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.321700 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.321806 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.322531 (+ 725us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.322637 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:50:04.323361 20644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:50:04.323473 20800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:04.325459 20724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53: No bootstrap required, opened a new log
W20260923 14:50:04.329221 20821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:50:04.329350 20821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
I20260923 14:50:04.329819 20809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } } }
I20260923 14:50:04.330425 20724 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } }
I20260923 14:50:04.330685 20809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [sys.catalog]: This master's current role is: FOLLOWER
Sep 23 14:50:04 dist-test-slave-wjf2 krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175004, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
I20260923 14:50:04.330854 20724 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac38f8d4e1194a12a5ee39b3e73caa53, State: Initialized, Role: FOLLOWER
I20260923 14:50:04.331236 20724 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } }
I20260923 14:50:04.332180 20823 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } } }
I20260923 14:50:04.332351 20823 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:04.332803 20724 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:50:04.334722 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.323247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.30.2.253:43531)
0923 14:50:04.323348 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.323353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.323542 (+ 189us) server_negotiation.cc:407] Connection header received
0923 14:50:04.323652 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.323654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.323704 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.323847 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.324734 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.325465 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.326588 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.326920 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.327008 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.330216 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.330220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.330223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.330255 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.332501 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.333009 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.333016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.333018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.333102 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.333408 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.333412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.333415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.333639 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.333827 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.334441 (+ 614us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.334560 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 14:50:04.334760 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.323972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:42177 (local address 127.0.0.1:46654)
0923 14:50:04.324173 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:04.324186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:04.324274 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:04.326046 (+ 1772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:04.326048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:04.326199 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:04.326290 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.326296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.327324 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.327330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.328249 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:04.328264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.329032 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.329038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:04.329135 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.329587 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:04.329605 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:04.331088 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:04.333286 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.333290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.333291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.333523 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.333789 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:04.333792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:04.333793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:04.333879 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:04.334319 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:04.334322 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:04.334415 (+ 93us) client_negotiation.cc:769] Sending connection context
0923 14:50:04.334494 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 14:50:04.334626 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260923 14:50:04.335359 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.324036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.30.2.252:42177)
0923 14:50:04.325850 (+ 1814us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.325855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.325868 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:50:04.325901 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.325904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.325939 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.326381 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.326470 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.327196 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.328401 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.328806 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.329343 (+ 537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.331251 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.331257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.331262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.331288 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.333173 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.333626 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.333629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.333630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.333703 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.333974 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.333976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.333979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.334205 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.334986 (+ 781us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.335072 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.335188 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1742}
I20260923 14:50:04.331683 20575 master_runner.cc:186] Error getting master registration for 127.30.2.253:43531: OK, instance_id { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" instance_seqno: 1790175004046840 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:50:04.335619 20575 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:50:04.338689 20800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6: Bootstrap starting.
W20260923 14:50:04.341290 20834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:50:04.341367 20834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:50:04.341974 20800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6: Neither blocks nor log segments found. Creating new log.
I20260923 14:50:04.342875 20800 log.cc:824] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6: Log is configured to *not* fsync() on all Append() calls
I20260923 14:50:04.341006 20651 master_runner.cc:186] Error getting master registration for 127.30.2.252:42177: OK, instance_id { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" instance_seqno: 1790175004285248 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:50:04.344310 20651 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:50:04.344503 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:50:04.345705 20617 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9d62b4b2edba41d89b520475c4529f7a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49420
I20260923 14:50:04.345932 20617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 14:50:04.346067 20617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:50:04.347113 20617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } }
I20260923 14:50:04.348495 20617 leader_election.cc:288] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [CANDIDATE]: Term 1 election: Requested vote from peers ac38f8d4e1194a12a5ee39b3e73caa53 (127.30.2.253:43531), b954a44e9bdf4d9599302d09fa0e3bb6 (127.30.2.252:42177)
I20260923 14:50:04.348511 20698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d62b4b2edba41d89b520475c4529f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53"
I20260923 14:50:04.348722 20698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:50:04.350037 20698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d62b4b2edba41d89b520475c4529f7a in term 1.
I20260923 14:50:04.350471 20594 leader_election.cc:302] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d62b4b2edba41d89b520475c4529f7a, ac38f8d4e1194a12a5ee39b3e73caa53; no voters:
I20260923 14:50:04.350760 20809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:50:04.350993 20809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [term 1 LEADER]: Becoming Leader. State: Replica: 9d62b4b2edba41d89b520475c4529f7a, State: Running, Role: LEADER
I20260923 14:50:04.351362 20809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } }
I20260923 14:50:04.354103 20842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d62b4b2edba41d89b520475c4529f7a. Latest consensus state: current_term: 1 leader_uuid: "9d62b4b2edba41d89b520475c4529f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } } }
I20260923 14:50:04.354292 20842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [sys.catalog]: This master's current role is: LEADER
I20260923 14:50:04.355063 20847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:50:04.356176 20847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:50:04.361660 20800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6: No bootstrap required, opened a new log
I20260923 14:50:04.365061 20698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [term 1 FOLLOWER]: Refusing update from remote peer 9d62b4b2edba41d89b520475c4529f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:50:04.365752 20842 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:50:04.368396 20800 raft_consensus.cc:348] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } }
I20260923 14:50:04.368651 20800 raft_consensus.cc:729] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b954a44e9bdf4d9599302d09fa0e3bb6, State: Initialized, Role: FOLLOWER
I20260923 14:50:04.368985 20800 consensus_queue.cc:260] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } }
I20260923 14:50:04.370255 20823 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d62b4b2edba41d89b520475c4529f7a. Latest consensus state: current_term: 1 leader_uuid: "9d62b4b2edba41d89b520475c4529f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } } }
I20260923 14:50:04.370383 20823 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:04.371184 20800 sys_catalog.cc:564] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:50:04.371420 20772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:50:04.371901 20773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d62b4b2edba41d89b520475c4529f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6"
I20260923 14:50:04.373461 20772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [term 1 FOLLOWER]: Refusing update from remote peer 9d62b4b2edba41d89b520475c4529f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:50:04.374086 20842 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:50:04.375095 20773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d62b4b2edba41d89b520475c4529f7a in term 1.
I20260923 14:50:04.384330 20842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d62b4b2edba41d89b520475c4529f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } } }
I20260923 14:50:04.384465 20842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [sys.catalog]: This master's current role is: LEADER
I20260923 14:50:04.385819 20823 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d62b4b2edba41d89b520475c4529f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } } }
I20260923 14:50:04.385928 20823 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:04.386632 20849 mvcc.cc:205] Tried to move back new op lower bound from 7332556817857929216 to 7332556817827135488. Current Snapshot: MvccSnapshot[applied={T|T < 7332556817857929216}]
I20260923 14:50:04.387117 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.350791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.30.2.254:46277)
0923 14:50:04.350919 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.350926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.351878 (+ 952us) server_negotiation.cc:407] Connection header received
0923 14:50:04.352006 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.352010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.352064 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.352147 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.353450 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.354268 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.355452 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.355846 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.355951 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.376277 (+ 20326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.376283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.376287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.376326 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.384754 (+ 8428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.385301 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.385305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.385307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.385408 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.385761 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.385765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.385767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.386035 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.386159 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.386864 (+ 705us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.386994 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 14:50:04.392048 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d62b4b2edba41d89b520475c4529f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } } }
I20260923 14:50:04.392198 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:04.392351 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.352355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.30.2.253:43531)
0923 14:50:04.352461 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.352465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.383296 (+ 30831us) server_negotiation.cc:407] Connection header received
0923 14:50:04.383432 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.383435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.383485 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.383575 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.384489 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.385287 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.386483 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.386828 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.386901 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.388711 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.388716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.388718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.388744 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.390418 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.390945 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.390948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.390950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.391040 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.391332 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.391335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.391337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.391550 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.391684 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.392072 (+ 388us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.392190 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260923 14:50:04.394105 20862 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:50:04.394196 20862 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:50:04.394385 20848 mvcc.cc:205] Tried to move back new op lower bound from 7332556817857929216 to 7332556817827135488. Current Snapshot: MvccSnapshot[applied={T|T < 7332556817857929216}]
I20260923 14:50:04.394654 20863 sys_catalog.cc:455] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d62b4b2edba41d89b520475c4529f7a. Latest consensus state: current_term: 1 leader_uuid: "9d62b4b2edba41d89b520475c4529f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } } }
I20260923 14:50:04.394784 20863 sys_catalog.cc:458] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:04.397984 20809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d62b4b2edba41d89b520475c4529f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } } }
I20260923 14:50:04.398136 20809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a [sys.catalog]: This master's current role is: LEADER
I20260923 14:50:04.399474 20863 sys_catalog.cc:455] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d62b4b2edba41d89b520475c4529f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d62b4b2edba41d89b520475c4529f7a" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 46277 } } peers { permanent_uuid: "ac38f8d4e1194a12a5ee39b3e73caa53" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 43531 } } peers { permanent_uuid: "b954a44e9bdf4d9599302d09fa0e3bb6" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 42177 } } }
I20260923 14:50:04.399634 20863 sys_catalog.cc:458] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:04.405107 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:04.351197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.30.2.252:42177)
0923 14:50:04.351304 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.351308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.377471 (+ 26163us) server_negotiation.cc:407] Connection header received
0923 14:50:04.377513 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.377517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.377574 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.377657 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.395733 (+ 18076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.396454 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.398491 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:04.398806 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:04.398887 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:04.401300 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:04.401305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:04.401308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:04.401335 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:04.403127 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.403654 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.403658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.403660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.403749 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:04.404090 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:04.404094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:04.404096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:04.404338 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:04.404454 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:04.404820 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 14:50:04.404940 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:50:04.407269 20847 catalog_manager.cc:1347] Generated new cluster ID: 36b2d7045f7a487fa6dad23907b2d633
I20260923 14:50:04.407333 20847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:50:04.432353 20847 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:50:04.434034 20847 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:50:04.447849 20847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d62b4b2edba41d89b520475c4529f7a: Generated new TSK 0
I20260923 14:50:04.451162 20847 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:50:04.489595 20728 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.487574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.30.2.253:43531)
0923 14:50:04.487726 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.487732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.488440 (+ 708us) server_negotiation.cc:407] Connection header received
0923 14:50:04.488673 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.488679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.488743 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.488801 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.489449 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39250: BlockingRecv error: recv got EOF from 127.0.0.1:39250 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 14:50:04.492173 20807 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.490289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.30.2.254:46277)
0923 14:50:04.490412 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.490418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.491147 (+ 729us) server_negotiation.cc:407] Connection header received
0923 14:50:04.491322 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.491325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.491381 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.491433 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.492039 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49446: BlockingRecv error: recv got EOF from 127.0.0.1:49446 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 14:50:04.493911 20806 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.490849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.30.2.252:42177)
0923 14:50:04.490968 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.490972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.492905 (+ 1933us) server_negotiation.cc:407] Connection header received
0923 14:50:04.493081 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.493084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.493134 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.493186 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.493777 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46660: BlockingRecv error: recv got EOF from 127.0.0.1:46660 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 14:50:04.498515 20728 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.496864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.30.2.253:43531)
0923 14:50:04.496974 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.496979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.497480 (+ 501us) server_negotiation.cc:407] Connection header received
0923 14:50:04.497711 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.497726 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.497807 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.497860 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.498404 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39252: BlockingRecv error: recv got EOF from 127.0.0.1:39252 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260923 14:50:04.503530 20728 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.501955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.30.2.253:43531)
0923 14:50:04.502059 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.502063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.502587 (+ 524us) server_negotiation.cc:407] Connection header received
0923 14:50:04.502789 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.502792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.502845 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.502897 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.503421 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39268: BlockingRecv error: recv got EOF from 127.0.0.1:39268 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 14:50:04.509472 20728 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.507772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.30.2.253:43531)
0923 14:50:04.507960 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.507964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.508481 (+ 517us) server_negotiation.cc:407] Connection header received
0923 14:50:04.508693 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.508698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.508771 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.508828 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.509366 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39276: BlockingRecv error: recv got EOF from 127.0.0.1:39276 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 14:50:04.517812 20728 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.515998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.30.2.253:43531)
0923 14:50:04.516157 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.516161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.516781 (+ 620us) server_negotiation.cc:407] Connection header received
0923 14:50:04.517013 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.517018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.517073 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.517127 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.517676 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39284: BlockingRecv error: recv got EOF from 127.0.0.1:39284 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260923 14:50:04.529814 20728 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.528120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.30.2.253:43531)
0923 14:50:04.528239 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.528244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.528867 (+ 623us) server_negotiation.cc:407] Connection header received
0923 14:50:04.529065 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.529069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.529120 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.529170 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.529683 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39296: BlockingRecv error: recv got EOF from 127.0.0.1:39296 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 14:50:04.550258 20728 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.548420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39306 (local address 127.30.2.253:43531)
0923 14:50:04.548550 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.548555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.549228 (+ 673us) server_negotiation.cc:407] Connection header received
0923 14:50:04.549494 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.549498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.549547 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.549640 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.550125 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39306: BlockingRecv error: recv got EOF from 127.0.0.1:39306 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 14:50:04.585726 20728 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.584362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39314 (local address 127.30.2.253:43531)
0923 14:50:04.584517 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.584522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.584787 (+ 265us) server_negotiation.cc:407] Connection header received
0923 14:50:04.584970 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.584974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.585030 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.585123 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.585598 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39314: BlockingRecv error: recv got EOF from 127.0.0.1:39314 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 14:50:04.653005 20728 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.652001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.30.2.253:43531)
0923 14:50:04.652146 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.652150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.652183 (+ 33us) server_negotiation.cc:407] Connection header received
0923 14:50:04.652322 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.652325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.652371 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.652446 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.652877 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39316: BlockingRecv error: recv got EOF from 127.0.0.1:39316 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260923 14:50:04.784621 20728 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:04.783409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39332 (local address 127.30.2.253:43531)
0923 14:50:04.783557 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:04.783561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:04.783841 (+ 280us) server_negotiation.cc:407] Connection header received
0923 14:50:04.783921 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:04.783924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:04.783976 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:04.784058 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:04.784517 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39332: BlockingRecv error: recv got EOF from 127.0.0.1:39332 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 14:50:05.044482 20728 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:05.043198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.30.2.253:43531)
0923 14:50:05.043374 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:05.043379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:05.043493 (+ 114us) server_negotiation.cc:407] Connection header received
0923 14:50:05.043701 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:05.043704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:05.043756 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:05.043872 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:05.044357 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39346: BlockingRecv error: recv got EOF from 127.0.0.1:39346 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 14:50:05.342788 20834 catalog_manager.cc:1259] Loaded cluster ID: 36b2d7045f7a487fa6dad23907b2d633
I20260923 14:50:05.342875 20834 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53: loading cluster ID for follower catalog manager: success
I20260923 14:50:05.344619 20834 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53: acquiring CA information for follower catalog manager: success
I20260923 14:50:05.345767 20834 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac38f8d4e1194a12a5ee39b3e73caa53: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:50:05.395606 20862 catalog_manager.cc:1259] Loaded cluster ID: 36b2d7045f7a487fa6dad23907b2d633
I20260923 14:50:05.395758 20862 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6: loading cluster ID for follower catalog manager: success
I20260923 14:50:05.397485 20862 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6: acquiring CA information for follower catalog manager: success
I20260923 14:50:05.398666 20862 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b954a44e9bdf4d9599302d09fa0e3bb6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:50:05.562970 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:05.558864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.30.2.253:43531)
0923 14:50:05.559145 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:05.559152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:05.559172 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:50:05.559360 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:05.559363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:05.559509 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:05.559628 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:50:05.559978 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:05.560531 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:05.561439 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:05.561670 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:05.561780 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:05.562096 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:50:05.562556 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:50:05.562691 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:05.562750 (+ 59us) server_negotiation.cc:299] Negotiation successful
0923 14:50:05.562807 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260923 14:50:05.569629 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:05.566144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46664 (local address 127.30.2.252:42177)
0923 14:50:05.566378 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:05.566382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:05.566409 (+ 27us) server_negotiation.cc:407] Connection header received
0923 14:50:05.566574 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:05.566576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:05.566691 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:05.566818 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:50:05.566993 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:05.567498 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:05.568292 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:05.568507 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:05.568583 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:05.568825 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:50:05.569238 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:50:05.569329 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:05.569386 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 14:50:05.569431 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
I20260923 14:50:05.577976 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:05.573724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46676 (local address 127.30.2.252:42177)
0923 14:50:05.573847 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:05.573852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:05.574193 (+ 341us) server_negotiation.cc:407] Connection header received
0923 14:50:05.574390 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:05.574402 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:05.574467 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:05.574527 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:50:05.574968 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:05.575584 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:05.576657 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:05.576971 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:05.577070 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:05.577371 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:50:05.577446 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:50:05.577510 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:05.577721 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 14:50:05.577791 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:50:05.578143 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:05.573435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39356 (local address 127.30.2.253:43531)
0923 14:50:05.573569 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:05.573575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:05.573770 (+ 195us) server_negotiation.cc:407] Connection header received
0923 14:50:05.573977 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:05.573980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:05.574031 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:05.574081 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:50:05.574465 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:05.575220 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:05.576793 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:05.577145 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:05.577224 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:05.577555 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:50:05.577658 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:50:05.577743 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:05.577923 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 14:50:05.577982 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:50:05.579221 20947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:05.573320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.30.2.254:46277)
0923 14:50:05.573761 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:05.573767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:05.573785 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:50:05.573891 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:05.573895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:05.574100 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:05.574299 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:50:05.574520 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:05.575202 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:05.577284 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:05.577672 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:05.577918 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:05.578263 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:50:05.578784 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:50:05.578876 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:05.578995 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 14:50:05.579045 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":62,"threads_started":1}
I20260923 14:50:05.582273 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 20575
I20260923 14:50:05.589879 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 20651
I20260923 14:50:05.596747 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 20730
2026-09-23T14:50:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3135 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:50:05 dist-test-slave-wjf2 krb5kdc[20952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:50:05 dist-test-slave-wjf2 krb5kdc[20952](info): set up 2 sockets
Sep 23 14:50:05 dist-test-slave-wjf2 krb5kdc[20952](info): commencing operation
krb5kdc: starting...
W20260923 14:50:06.633652 30731 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/test-user.keytab.
Sep 23 14:50:06 dist-test-slave-wjf2 krb5kdc[20952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
Entry for principal kudu/127.30.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab.
I20260923 14:50:06.763546 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:33893
--webserver_interface=127.30.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:39761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:33893,127.30.2.253:38757,127.30.2.252:32935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:50:06.876055 20965 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:50:06.876430 20965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:50:06.876504 20965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:50:06.876585 20965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:50:06.880630 20965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:50:06.880728 20965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:50:06.880767 20965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:50:06.880801 20965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:50:06.880837 20965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:50:06.886190 20965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:39761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:33893,127.30.2.253:38757,127.30.2.252:32935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.254.keytab
--principal=kudu/127.30.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.254:33893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:50:06.887360 20965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:50:06.888474 20965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:50:06.897554 20971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:06.897604 20970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:50:06.897652 20965 server_base.cc:1034] running on GCE node
W20260923 14:50:06.897574 20973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:50:06.898420 20965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:50:06.899688 20965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:50:06.900871 20965 hybrid_clock.cc:648] HybridClock initialized: now 1790175006900852 us; error 35 us; skew 500 ppm
Sep 23 14:50:06 dist-test-slave-wjf2 krb5kdc[20952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175006, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:50:06.903883 20965 init.cc:377] Logged in from keytab as kudu/127.30.2.254@KRBTEST.COM (short username kudu)
I20260923 14:50:06.904877 20965 webserver.cc:460] Webserver started at http://127.30.2.254:41837/ using document root <none> and password file <none>
I20260923 14:50:06.905493 20965 fs_manager.cc:359] Metadata directory not provided
I20260923 14:50:06.905581 20965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:50:06.905828 20965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:50:06.908332 20965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/data/instance:
uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d"
format_stamp: "Formatted at 2026-09-23 14:50:06 on dist-test-slave-wjf2"
server_key: "93132d79de5867b22e25ff3882bb4a24"
server_key_iv: "d870a1669e20436e6213ec9c844db74d"
server_key_version: "encryptionkey@0"
I20260923 14:50:06.908983 20965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/wal/instance:
uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d"
format_stamp: "Formatted at 2026-09-23 14:50:06 on dist-test-slave-wjf2"
server_key: "93132d79de5867b22e25ff3882bb4a24"
server_key_iv: "d870a1669e20436e6213ec9c844db74d"
server_key_version: "encryptionkey@0"
I20260923 14:50:06.913022 20965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 14:50:06.915583 20980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:50:06.916936 20965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:50:06.917083 20965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/wal
uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d"
format_stamp: "Formatted at 2026-09-23 14:50:06 on dist-test-slave-wjf2"
server_key: "93132d79de5867b22e25ff3882bb4a24"
server_key_iv: "d870a1669e20436e6213ec9c844db74d"
server_key_version: "encryptionkey@0"
I20260923 14:50:06.917207 20965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:50:06.938813 20965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:50:06.941919 20965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:50:06.942121 20965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:50:06.949155 20965 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.254:33893
I20260923 14:50:06.949457 21033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.254:33893 every 8 connection(s)
I20260923 14:50:06.950157 20965 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/data/info.pb
I20260923 14:50:06.952999 21034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:50:06.956809 21034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:06.960052 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 20965
I20260923 14:50:06.960170 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-0/wal/instance
I20260923 14:50:06.960459 30731 external_mini_cluster.cc:1428] Setting key b9390753f4724d98040fd512a891600e
Sep 23 14:50:06 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175006, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.2.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.253@KRBTEST.COM" created.
I20260923 14:50:06.972714 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:06.959058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.30.2.254:33893)
0923 14:50:06.959497 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:06.959512 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:06.959678 (+ 166us) server_negotiation.cc:407] Connection header received
0923 14:50:06.960868 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:06.960879 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:06.961130 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:06.961353 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:06.962202 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:06.963116 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:06.964742 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:06.965134 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:06.965266 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:06.968007 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:06.968014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:06.968028 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:06.968055 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:06.969782 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:06.970286 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:06.970291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:06.970292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:06.970394 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:06.970674 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:06.970676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:06.970677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:06.971035 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:06.971136 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:06.971543 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 14:50:06.971751 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":304,"spinlock_wait_cycles":34688,"thread_start_us":173,"threads_started":1}
I20260923 14:50:06.972761 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:06.959059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33893 (local address 127.0.0.1:34588)
0923 14:50:06.959524 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:06.959573 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:06.960538 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:06.961348 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:06.961352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:06.961671 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:06.962011 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:06.962026 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:06.963261 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:06.963267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:06.964578 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:06.964593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:06.965310 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:06.965317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:06.965468 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:06.966080 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:06.966108 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:06.967852 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:06.969890 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:06.969898 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:06.969906 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:06.970134 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:06.970476 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:06.970478 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:06.970480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:06.970591 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:06.971133 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:06.971137 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:06.971309 (+ 172us) client_negotiation.cc:769] Sending connection context
0923 14:50:06.971448 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 14:50:06.971878 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":161,"thread_start_us":140,"threads_started":1}
I20260923 14:50:06.975283 21034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:06.976085 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:06.975769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:38757 (local address 127.0.0.1:35324)
0923 14:50:06.975906 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:06.975997 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260923 14:50:06.976372 20984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.2.253:38757: connect: Connection refused (error 111)
I20260923 14:50:06.977329 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:06.977089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:38757 (local address 127.0.0.1:35326)
0923 14:50:06.977220 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:06.977271 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 14:50:06.977568 21034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.253:38757: Network error: Client connection negotiation failed: client connection to 127.30.2.253:38757: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab.
Entry for principal kudu/127.30.2.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab.
I20260923 14:50:06.989280 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.253:38757
--webserver_interface=127.30.2.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab
--principal=kudu/127.30.2.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:39761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:33893,127.30.2.253:38757,127.30.2.252:32935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:50:07.026870 21034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } attempt: 1
I20260923 14:50:07.027917 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.027503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:38757 (local address 127.0.0.1:35340)
0923 14:50:07.027698 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.027768 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 14:50:07.028748 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.028504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:38757 (local address 127.0.0.1:35352)
0923 14:50:07.028625 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.028681 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 14:50:07.028893 21034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.253:38757: Network error: Client connection negotiation failed: client connection to 127.30.2.253:38757: connect: Connection refused (error 111)
I20260923 14:50:07.097064 21034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } attempt: 2
I20260923 14:50:07.097918 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.097571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:38757 (local address 127.0.0.1:35368)
0923 14:50:07.097775 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.097837 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 14:50:07.098618 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.098406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:38757 (local address 127.0.0.1:35380)
0923 14:50:07.098512 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.098561 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.253:38757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 14:50:07.098743 21034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.253:38757: Network error: Client connection negotiation failed: client connection to 127.30.2.253:38757: connect: Connection refused (error 111)
W20260923 14:50:07.104156 21041 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:50:07.104483 21041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:50:07.104539 21041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:50:07.104619 21041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:50:07.108765 21041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:50:07.108861 21041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:50:07.108901 21041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:50:07.108955 21041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:50:07.108994 21041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:50:07.114656 21041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:39761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:33893,127.30.2.253:38757,127.30.2.252:32935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.253.keytab
--principal=kudu/127.30.2.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.253:38757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.2.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:50:07.115887 21041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:50:07.116942 21041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:50:07.125625 21047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:07.125651 21049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:07.125639 21046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:50:07.126088 21041 server_base.cc:1034] running on GCE node
I20260923 14:50:07.126662 21041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:50:07.127863 21041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:50:07.129040 21041 hybrid_clock.cc:648] HybridClock initialized: now 1790175007129019 us; error 28 us; skew 500 ppm
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175007, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:50:07.132017 21041 init.cc:377] Logged in from keytab as kudu/127.30.2.253@KRBTEST.COM (short username kudu)
I20260923 14:50:07.132925 21041 webserver.cc:460] Webserver started at http://127.30.2.253:38507/ using document root <none> and password file <none>
I20260923 14:50:07.133530 21041 fs_manager.cc:359] Metadata directory not provided
I20260923 14:50:07.133620 21041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:50:07.133854 21041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:50:07.136006 21041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/data/instance:
uuid: "636bf2fce43648ad871cf4cb37b7a6d0"
format_stamp: "Formatted at 2026-09-23 14:50:07 on dist-test-slave-wjf2"
server_key: "a6c68109ee5a9309ccd98bb547193144"
server_key_iv: "8c32d1ab55c5e3ac976a738c9aa7cb18"
server_key_version: "encryptionkey@0"
I20260923 14:50:07.136567 21041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/wal/instance:
uuid: "636bf2fce43648ad871cf4cb37b7a6d0"
format_stamp: "Formatted at 2026-09-23 14:50:07 on dist-test-slave-wjf2"
server_key: "a6c68109ee5a9309ccd98bb547193144"
server_key_iv: "8c32d1ab55c5e3ac976a738c9aa7cb18"
server_key_version: "encryptionkey@0"
I20260923 14:50:07.140365 21041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 14:50:07.142766 21056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:50:07.144073 21041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:50:07.144217 21041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/wal
uuid: "636bf2fce43648ad871cf4cb37b7a6d0"
format_stamp: "Formatted at 2026-09-23 14:50:07 on dist-test-slave-wjf2"
server_key: "a6c68109ee5a9309ccd98bb547193144"
server_key_iv: "8c32d1ab55c5e3ac976a738c9aa7cb18"
server_key_version: "encryptionkey@0"
I20260923 14:50:07.144337 21041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:50:07.159181 21041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:50:07.162488 21041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:50:07.162709 21041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:50:07.170043 21041 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.253:38757
I20260923 14:50:07.170336 21108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.253:38757 every 8 connection(s)
I20260923 14:50:07.171167 21041 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/data/info.pb
I20260923 14:50:07.174105 21109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:50:07.175925 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 21041
I20260923 14:50:07.176051 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-1/wal/instance
I20260923 14:50:07.176319 30731 external_mini_cluster.cc:1428] Setting key 8cecab23c470b923e6f3a19f6d331b6e
I20260923 14:50:07.178133 21109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.2.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.2.252@KRBTEST.COM" created.
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175007, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:50:07.189970 21034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } attempt: 3
I20260923 14:50:07.195964 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.180113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.30.2.254:33893)
0923 14:50:07.180283 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.180288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.181174 (+ 886us) server_negotiation.cc:407] Connection header received
0923 14:50:07.182135 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.182139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.182206 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.182323 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.183529 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.184321 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.185859 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.186179 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.186284 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.190099 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.190107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.190111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.190157 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.193110 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.193836 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.193840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.193842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.193938 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.194346 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.194350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.194353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.194633 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.194736 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.195576 (+ 840us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.195732 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 14:50:07.197774 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.180521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33893 (local address 127.0.0.1:34600)
0923 14:50:07.180996 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.181050 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:07.181907 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:07.182510 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:07.182518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:07.182884 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:07.183365 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.183378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.184462 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.184466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.185608 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.185620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.186373 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.186377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.186601 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.187407 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:07.187426 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:07.189747 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:07.193234 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.193242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.193272 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.193718 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.194042 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.194046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.194048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.194229 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.194765 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:07.194776 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:07.195217 (+ 441us) client_negotiation.cc:769] Sending connection context
0923 14:50:07.195501 (+ 284us) client_negotiation.cc:240] Negotiation successful
0923 14:50:07.195996 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":143,"threads_started":1}
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175006, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
I20260923 14:50:07.200644 21109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:07.203132 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.190940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:38757 (local address 127.0.0.1:35384)
0923 14:50:07.191174 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.191193 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:07.191315 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:07.192375 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:07.192380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:07.192638 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:07.192806 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.192817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.194538 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.194546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.195189 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.195199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.196111 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.196117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.196235 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.196839 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:07.196861 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:07.198304 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:07.201102 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.201106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.201108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.201414 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.201824 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.201827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.201828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.201929 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.202553 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:07.202556 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:07.202666 (+ 110us) client_negotiation.cc:769] Sending connection context
0923 14:50:07.202785 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 14:50:07.202943 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 14:50:07.203213 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.190700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.30.2.253:38757)
0923 14:50:07.191438 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.191456 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.191494 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:50:07.191580 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.191591 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.191936 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.192122 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.192976 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.194393 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.195323 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.195947 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.196493 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.198476 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.198490 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.198511 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.198540 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.200973 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.201570 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.201577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.201579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.201679 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.202024 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.202029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.202031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.202454 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.202577 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.202881 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.203034 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":530,"thread_start_us":112,"threads_started":1}
I20260923 14:50:07.205315 21034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:07.206133 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.205872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58472)
0923 14:50:07.205995 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.206051 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.2.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab.
Entry for principal kudu/127.30.2.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab.
I20260923 14:50:07.207039 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.206686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58482)
0923 14:50:07.206908 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.206960 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260923 14:50:07.207170 21034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:32935: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175007, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
I20260923 14:50:07.207995 30731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
/tmp/dist-test-tasknhhgz8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.252:32935
--webserver_interface=127.30.2.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab
--principal=kudu/127.30.2.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.2.212:39761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.2.254:33893,127.30.2.253:38757,127.30.2.252:32935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:50:07.211513 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.201372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:38757 (local address 127.0.0.1:35396)
0923 14:50:07.201521 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.201537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:07.201608 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:07.202083 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:07.202092 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:07.202263 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:07.202380 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.202388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.203440 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.203444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.204255 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.204265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.205356 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.205359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.205512 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.206320 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:07.206334 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:07.207878 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:07.209981 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.209984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.209986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.210184 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.210487 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.210492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.210494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.210575 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.210967 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:07.210970 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:07.211059 (+ 89us) client_negotiation.cc:769] Sending connection context
0923 14:50:07.211162 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 14:50:07.211328 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 14:50:07.211546 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.201426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.30.2.253:38757)
0923 14:50:07.201818 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.201824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.201836 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:50:07.201865 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.201867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.201969 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.202065 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.202555 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.203099 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.204379 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.205162 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.205356 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.208031 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.208041 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.208043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.208064 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.209842 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.210301 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.210305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.210307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.210397 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.210665 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.210668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.210669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.210868 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.210967 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.211207 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.211402 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":330,"thread_start_us":200,"threads_started":1}
I20260923 14:50:07.212488 21109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:07.213316 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.213037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58490)
0923 14:50:07.213163 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.213241 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 14:50:07.213537 21058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
I20260923 14:50:07.214452 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.214232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58502)
0923 14:50:07.214345 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.214391 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 14:50:07.214612 21109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:32935: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
I20260923 14:50:07.238332 21034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } attempt: 1
I20260923 14:50:07.239459 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.239003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58516)
0923 14:50:07.239243 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.239344 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 14:50:07.240527 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.240211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58532)
0923 14:50:07.240358 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.240442 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 14:50:07.240667 21034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:32935: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
I20260923 14:50:07.263906 21109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } attempt: 1
I20260923 14:50:07.264870 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.264487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58548)
0923 14:50:07.264681 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.264767 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 14:50:07.265722 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.265458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58554)
0923 14:50:07.265585 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.265646 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 14:50:07.266005 21109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:32935: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
I20260923 14:50:07.315850 21034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } attempt: 2
I20260923 14:50:07.316828 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.316492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58556)
0923 14:50:07.316678 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.316747 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 14:50:07.317557 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.317346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58564)
0923 14:50:07.317453 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.317502 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 14:50:07.317682 21034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:32935: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
I20260923 14:50:07.334299 21109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } attempt: 2
I20260923 14:50:07.335299 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.334902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58578)
0923 14:50:07.335091 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.335183 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 14:50:07.336207 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.335942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58590)
0923 14:50:07.336061 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.336124 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 14:50:07.336354 21109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.2.252:32935: Network error: Client connection negotiation failed: client connection to 127.30.2.252:32935: connect: Connection refused (error 111)
W20260923 14:50:07.344858 21116 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:50:07.345192 21116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:50:07.345217 21116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:50:07.345301 21116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:50:07.349869 21116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:50:07.349941 21116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:50:07.349961 21116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:50:07.349979 21116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:50:07.349995 21116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:50:07.355860 21116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.2.212:39761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.2.254:33893,127.30.2.253:38757,127.30.2.252:32935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/krb5kdc/kudu_127.30.2.252.keytab
--principal=kudu/127.30.2.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.2.252:32935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.2.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:50:07.357062 21116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:50:07.358210 21116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:50:07.367343 21116 server_base.cc:1034] running on GCE node
W20260923 14:50:07.367298 21123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:07.367291 21122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:50:07.367285 21125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:50:07.368227 21116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:50:07.369443 21116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:50:07.370601 21116 hybrid_clock.cc:648] HybridClock initialized: now 1790175007370560 us; error 53 us; skew 500 ppm
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175007, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:50:07.373646 21116 init.cc:377] Logged in from keytab as kudu/127.30.2.252@KRBTEST.COM (short username kudu)
I20260923 14:50:07.374652 21116 webserver.cc:460] Webserver started at http://127.30.2.252:44485/ using document root <none> and password file <none>
I20260923 14:50:07.375298 21116 fs_manager.cc:359] Metadata directory not provided
I20260923 14:50:07.375396 21116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:50:07.375636 21116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:50:07.377863 21116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/data/instance:
uuid: "e748f750acc8469d8dd19d3245612831"
format_stamp: "Formatted at 2026-09-23 14:50:07 on dist-test-slave-wjf2"
server_key: "020dcb5d8e9789400eda8d6f18083c17"
server_key_iv: "b62bbe6ec66462c4c6005a959ba7aa12"
server_key_version: "encryptionkey@0"
I20260923 14:50:07.378463 21116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/wal/instance:
uuid: "e748f750acc8469d8dd19d3245612831"
format_stamp: "Formatted at 2026-09-23 14:50:07 on dist-test-slave-wjf2"
server_key: "020dcb5d8e9789400eda8d6f18083c17"
server_key_iv: "b62bbe6ec66462c4c6005a959ba7aa12"
server_key_version: "encryptionkey@0"
I20260923 14:50:07.382580 21116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 14:50:07.385293 21132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:50:07.386647 21116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 14:50:07.386827 21116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/data,/tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/wal
uuid: "e748f750acc8469d8dd19d3245612831"
format_stamp: "Formatted at 2026-09-23 14:50:07 on dist-test-slave-wjf2"
server_key: "020dcb5d8e9789400eda8d6f18083c17"
server_key_iv: "b62bbe6ec66462c4c6005a959ba7aa12"
server_key_version: "encryptionkey@0"
I20260923 14:50:07.386951 21116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:50:07.402191 21116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:50:07.405280 21116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:50:07.405500 21116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:50:07.412897 21116 rpc_server.cc:307] RPC server started. Bound to: 127.30.2.252:32935
I20260923 14:50:07.413182 21185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.2.252:32935 every 8 connection(s)
I20260923 14:50:07.413997 21116 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/data/info.pb
I20260923 14:50:07.416110 30731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu as pid 21116
I20260923 14:50:07.416249 30731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknhhgz8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174723338690-30731-0/minicluster-data/master-2/wal/instance
I20260923 14:50:07.416560 30731 external_mini_cluster.cc:1428] Setting key 2827e177a4bda36a24f0a7453222163d
I20260923 14:50:07.416834 21034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } attempt: 3
I20260923 14:50:07.417750 21186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:50:07.422861 21186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175006, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.254@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
I20260923 14:50:07.427520 21109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } attempt: 3
I20260923 14:50:07.431890 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.417259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58604)
0923 14:50:07.417419 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.417436 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:07.417537 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:07.420110 (+ 2573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:07.420113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:07.420325 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:07.420452 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.420460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.422162 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.422167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.423011 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.423024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.423979 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.423983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.424075 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.424652 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:07.424671 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:07.426214 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:07.429573 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.429576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.429579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.429844 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.430279 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.430282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.430285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.430393 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.431288 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:07.431291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:07.431400 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 14:50:07.431498 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 14:50:07.431644 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175007, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175007, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.253@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
I20260923 14:50:07.433634 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.417993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.30.2.252:32935)
0923 14:50:07.418489 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.418509 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.418566 (+ 57us) server_negotiation.cc:407] Connection header received
0923 14:50:07.419324 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.419342 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.419760 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.420142 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.420939 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.421982 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.423181 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.423855 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.424002 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.427061 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.427064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.427067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.427091 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.429442 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.429973 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.429979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.429986 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.430165 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.430500 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.430503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.430505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.431150 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.431359 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.432207 (+ 848us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.432523 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":173,"server-negotiator.queue_time_us":323,"spinlock_wait_cycles":29568,"thread_start_us":145,"threads_started":1}
I20260923 14:50:07.433634 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.418646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.30.2.252:32935)
0923 14:50:07.418935 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.418945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.418960 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:50:07.419340 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.419348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.419762 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.420575 (+ 813us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.420709 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.422061 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.423141 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.423846 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.424000 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.426378 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.426384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.426408 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.426443 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.429457 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.429973 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.429979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.429986 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.430165 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.430502 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.430504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.430506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.431150 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.432399 (+ 1249us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.432470 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.432613 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":290,"server-negotiator.queue_time_us":218,"spinlock_wait_cycles":52608,"thread_start_us":132,"threads_started":1}
I20260923 14:50:07.437592 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.425778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.30.2.254:33893)
0923 14:50:07.425943 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.425948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.425964 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:50:07.426008 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.426012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.426057 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.426142 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.427034 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.427735 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.429179 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.429519 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.429591 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.433168 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.433171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.433174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.433203 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.435266 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.436177 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.436181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.436182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.436269 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.436591 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.436594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.436596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.436855 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.436953 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.437370 (+ 417us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.437474 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 14:50:07.438149 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.424890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.254:33893 (local address 127.0.0.1:34602)
0923 14:50:07.425331 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.425392 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:07.425517 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:07.426220 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:07.426227 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:07.426692 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:07.426884 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.426895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.427881 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.427884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.429053 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.429061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.429755 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.429759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.429880 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.430978 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:07.430998 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:07.433014 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:07.435720 (+ 2706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.435723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.435727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.436033 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.436360 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.436365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.436367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.436475 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.437007 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:07.437015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:07.437143 (+ 128us) client_negotiation.cc:769] Sending connection context
0923 14:50:07.437741 (+ 598us) client_negotiation.cc:240] Negotiation successful
0923 14:50:07.437961 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":118,"threads_started":1}
I20260923 14:50:07.438668 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d: Bootstrap starting.
I20260923 14:50:07.440797 21186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:07.441546 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.428007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58610)
0923 14:50:07.428180 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.428196 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:07.428283 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:07.428584 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:07.428588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:07.428765 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:07.428876 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.428882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.429934 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.429937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.430883 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.430899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.431571 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.431576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.431672 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.432131 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:07.432146 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:07.433530 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:07.435512 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.435516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.435518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.435766 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.440337 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.440341 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.440343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.440447 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.440965 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:07.440971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:07.441084 (+ 113us) client_negotiation.cc:769] Sending connection context
0923 14:50:07.441180 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 14:50:07.441356 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 14:50:07.441636 21034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d: Neither blocks nor log segments found. Creating new log.
I20260923 14:50:07.442536 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.428043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.30.2.252:32935)
0923 14:50:07.428341 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.428350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.428363 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:50:07.428396 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.428398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.428438 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.428988 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.429072 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.429791 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.431040 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.431418 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.431925 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.433689 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.433695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.433699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.433724 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.435404 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.440106 (+ 4702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.440110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.440111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.440211 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.440574 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.440578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.440579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.440780 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.442109 (+ 1329us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.442184 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.442348 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":100,"threads_started":1}
I20260923 14:50:07.442595 21034 log.cc:824] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d: Log is configured to *not* fsync() on all Append() calls
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.254@KRBTEST.COM
I20260923 14:50:07.444854 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d: No bootstrap required, opened a new log
I20260923 14:50:07.446259 21109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0: Bootstrap starting.
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175007, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
I20260923 14:50:07.448460 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.436716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.30.2.254:33893)
0923 14:50:07.437003 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.437015 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.437027 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:50:07.437107 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.437110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.437164 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.437254 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.437981 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.438810 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.439858 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.440475 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.440570 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.443584 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.443589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.443592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.443618 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.445673 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.446152 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.446156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.446157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.446246 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.446593 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.446596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.446598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.446801 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.446966 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.447692 (+ 726us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.447843 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260923 14:50:07.449714 21109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0: Neither blocks nor log segments found. Creating new log.
I20260923 14:50:07.449579 21034 raft_consensus.cc:348] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } }
I20260923 14:50:07.449824 21034 raft_consensus.cc:729] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede4e20c4faa48d78cd7b53fefcbcd8d, State: Initialized, Role: FOLLOWER
I20260923 14:50:07.450189 21034 consensus_queue.cc:260] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } }
I20260923 14:50:07.450699 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.441537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.30.2.253:38757)
0923 14:50:07.441655 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.441660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.441830 (+ 170us) server_negotiation.cc:407] Connection header received
0923 14:50:07.442004 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.442007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.442048 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.442094 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.442696 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.443418 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.444466 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.444794 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.444882 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.447104 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.447109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.447116 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.447141 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.448831 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.449318 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.449321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.449323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.449405 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.449745 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.449749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.449750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.449986 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.450067 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.450447 (+ 380us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.450560 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:50:07.451169 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.441290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.253:38757 (local address 127.0.0.1:35406)
0923 14:50:07.441761 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.441775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:07.441901 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:07.442216 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:07.442219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:07.442447 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:07.442558 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.442565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.443557 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.443560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.444331 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.444342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.445030 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.445033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.445119 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.445535 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:07.445549 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:07.446914 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:07.448947 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.448950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.448952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.449187 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.449521 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.449527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.449532 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.449627 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.450153 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:07.450156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:07.450249 (+ 93us) client_negotiation.cc:769] Sending connection context
0923 14:50:07.450835 (+ 586us) client_negotiation.cc:240] Negotiation successful
0923 14:50:07.450980 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402}
I20260923 14:50:07.451637 21109 log.cc:824] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0: Log is configured to *not* fsync() on all Append() calls
I20260923 14:50:07.451828 21034 sys_catalog.cc:564] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:50:07.451962 21186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:50:07.453848 21195 sys_catalog.cc:455] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } } }
I20260923 14:50:07.454015 21195 sys_catalog.cc:458] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:07.456038 21109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0: No bootstrap required, opened a new log
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.2.253@KRBTEST.COM
Sep 23 14:50:07 dist-test-slave-wjf2 krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790175007, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.2.252@KRBTEST.COM for kudu/127.30.2.252@KRBTEST.COM
I20260923 14:50:07.460048 21109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } }
I20260923 14:50:07.460259 21109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636bf2fce43648ad871cf4cb37b7a6d0, State: Initialized, Role: FOLLOWER
W20260923 14:50:07.460337 21207 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:50:07.460397 21207 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:50:07.460602 21109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } }
I20260923 14:50:07.457221 20965 master_runner.cc:186] Error getting master registration for 127.30.2.253:38757: OK, instance_id { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" instance_seqno: 1790175007167913 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:50:07.461715 20965 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:50:07.461783 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.450314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.30.2.253:38757)
0923 14:50:07.450445 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.450449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.450624 (+ 175us) server_negotiation.cc:407] Connection header received
0923 14:50:07.450791 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.450794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.450838 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.450884 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.451767 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.452573 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.453670 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.454023 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.454112 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.458075 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.458079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.458082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.458110 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.459830 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.460330 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.460333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.460335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.460426 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.460736 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.460740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.460741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.460962 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.461124 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.461499 (+ 375us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.461626 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 14:50:07.462056 21109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:50:07.466094 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.452421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.2.252:32935 (local address 127.0.0.1:58614)
0923 14:50:07.452508 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0923 14:50:07.452523 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:50:07.452596 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:50:07.452837 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:50:07.452839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:50:07.452961 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:50:07.453026 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.453050 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.453922 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.453925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.454816 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:50:07.454829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.455599 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.455603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:50:07.455685 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.456200 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:50:07.456219 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:50:07.462488 (+ 6269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:50:07.464486 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.464490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.464492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.464725 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.465022 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:50:07.465025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:50:07.465027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:50:07.465107 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:50:07.465560 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:50:07.465562 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:50:07.465655 (+ 93us) client_negotiation.cc:769] Sending connection context
0923 14:50:07.465759 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 14:50:07.465900 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 14:50:07.466679 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.452494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.30.2.252:32935)
0923 14:50:07.452597 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.452604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.452619 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:50:07.452656 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.452662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.452708 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.452784 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.453165 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.453816 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.454946 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.455370 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.455481 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.462641 (+ 7160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.462645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.462648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.462674 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.464366 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.464838 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.464843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.464845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.464927 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.465202 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.465205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.465206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.465437 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.465528 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.466418 (+ 890us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.466541 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:50:07.468837 30731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 14:50:07.469151 21220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:50:07.469221 21220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:50:07.469846 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831: Bootstrap starting.
I20260923 14:50:07.471625 21008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34610
I20260923 14:50:07.471894 21008 raft_consensus.cc:482] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 14:50:07.472002 21008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:50:07.472702 21186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831: Neither blocks nor log segments found. Creating new log.
I20260923 14:50:07.473413 21008 raft_consensus.cc:504] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } }
I20260923 14:50:07.473584 21186 log.cc:824] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831: Log is configured to *not* fsync() on all Append() calls
I20260923 14:50:07.474795 21083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "636bf2fce43648ad871cf4cb37b7a6d0"
I20260923 14:50:07.475021 21083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:50:07.476511 21083 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ede4e20c4faa48d78cd7b53fefcbcd8d in term 1.
I20260923 14:50:07.476912 20984 leader_election.cc:302] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 636bf2fce43648ad871cf4cb37b7a6d0, ede4e20c4faa48d78cd7b53fefcbcd8d; no voters:
I20260923 14:50:07.477190 21195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:50:07.477425 21195 raft_consensus.cc:686] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [term 1 LEADER]: Becoming Leader. State: Replica: ede4e20c4faa48d78cd7b53fefcbcd8d, State: Running, Role: LEADER
I20260923 14:50:07.477782 21195 consensus_queue.cc:237] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } }
I20260923 14:50:07.476263 21041 master_runner.cc:186] Error getting master registration for 127.30.2.252:32935: OK, instance_id { permanent_uuid: "e748f750acc8469d8dd19d3245612831" instance_seqno: 1790175007410841 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:50:07.479614 21041 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:50:07.479952 21008 leader_election.cc:288] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [CANDIDATE]: Term 1 election: Requested vote from peers 636bf2fce43648ad871cf4cb37b7a6d0 (127.30.2.253:38757), e748f750acc8469d8dd19d3245612831 (127.30.2.252:32935)
I20260923 14:50:07.465843 21209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } } }
I20260923 14:50:07.482379 21209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:07.484300 21223 sys_catalog.cc:455] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ede4e20c4faa48d78cd7b53fefcbcd8d. Latest consensus state: current_term: 1 leader_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } } }
I20260923 14:50:07.484438 21223 sys_catalog.cc:458] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [sys.catalog]: This master's current role is: LEADER
I20260923 14:50:07.486037 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831: No bootstrap required, opened a new log
I20260923 14:50:07.490279 21231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:50:07.491695 21231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:50:07.493001 21186 raft_consensus.cc:348] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } }
I20260923 14:50:07.493737 21186 raft_consensus.cc:729] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e748f750acc8469d8dd19d3245612831, State: Initialized, Role: FOLLOWER
I20260923 14:50:07.494275 21186 consensus_queue.cc:260] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } }
I20260923 14:50:07.497048 21160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e748f750acc8469d8dd19d3245612831"
I20260923 14:50:07.497301 21160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:50:07.498863 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.482709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.30.2.254:33893)
0923 14:50:07.482851 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.482858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.485248 (+ 2390us) server_negotiation.cc:407] Connection header received
0923 14:50:07.485455 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.485458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.485509 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.485560 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.486699 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.487433 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.488619 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.488984 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.489066 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.492927 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.492931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.492934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.492968 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.496004 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.497293 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.497299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.497301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.497402 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.497795 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.497798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.497800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.498033 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.498147 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.498496 (+ 349us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.498629 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:50:07.498886 21160 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ede4e20c4faa48d78cd7b53fefcbcd8d in term 1.
I20260923 14:50:07.500561 21186 sys_catalog.cc:564] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:50:07.507561 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.483068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.30.2.253:38757)
0923 14:50:07.483179 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.483184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.489838 (+ 6654us) server_negotiation.cc:407] Connection header received
0923 14:50:07.490889 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.490893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.491026 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.491097 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.493336 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.494122 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.499904 (+ 5782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.500207 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.500295 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.502238 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.502242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.502245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.502274 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.505605 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.506098 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.506102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.506104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.506202 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.506548 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.506552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.506554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.506792 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.506907 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.507279 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.507409 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:50:07.508138 21160 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [term 1 FOLLOWER]: Refusing update from remote peer ede4e20c4faa48d78cd7b53fefcbcd8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:50:07.508783 21223 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:50:07.510560 21083 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [term 1 FOLLOWER]: Refusing update from remote peer ede4e20c4faa48d78cd7b53fefcbcd8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:50:07.511099 21223 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:50:07.514338 21233 sys_catalog.cc:455] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } } }
I20260923 14:50:07.517786 21233 sys_catalog.cc:458] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:07.517956 21233 sys_catalog.cc:455] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ede4e20c4faa48d78cd7b53fefcbcd8d. Latest consensus state: current_term: 1 leader_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } } }
I20260923 14:50:07.518128 21233 sys_catalog.cc:458] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 14:50:07.517691 21244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:50:07.518318 21244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:50:07.520234 21209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ede4e20c4faa48d78cd7b53fefcbcd8d. Latest consensus state: current_term: 1 leader_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } } }
I20260923 14:50:07.520377 21209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:07.521929 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:07.484787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.30.2.252:32935)
0923 14:50:07.484906 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.484911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.490525 (+ 5614us) server_negotiation.cc:407] Connection header received
0923 14:50:07.490711 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.490715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.490782 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.490835 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.503248 (+ 12413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.504055 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.511708 (+ 7653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:07.512069 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:07.512177 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:07.517518 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:50:07.517522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:50:07.517525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:50:07.517552 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:50:07.519266 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.519801 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.519805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.519807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.519902 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:50:07.520263 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:50:07.520267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:50:07.520269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:50:07.520677 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:50:07.520839 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:07.521581 (+ 742us) server_negotiation.cc:299] Negotiation successful
0923 14:50:07.521742 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:50:07.529312 21223 sys_catalog.cc:455] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } } }
I20260923 14:50:07.529453 21223 sys_catalog.cc:458] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [sys.catalog]: This master's current role is: LEADER
I20260923 14:50:07.531026 21247 mvcc.cc:205] Tried to move back new op lower bound from 7332556830744944640 to 7332556830632493056. Current Snapshot: MvccSnapshot[applied={T|T < 7332556830744944640}]
I20260923 14:50:07.531890 21246 sys_catalog.cc:455] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } } }
I20260923 14:50:07.532013 21246 sys_catalog.cc:458] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:07.536286 21231 catalog_manager.cc:1347] Generated new cluster ID: 28dfa0f5983344ffbf0d51715e2596f1
I20260923 14:50:07.536365 21231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:50:07.536355 21195 sys_catalog.cc:455] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } } }
I20260923 14:50:07.536502 21195 sys_catalog.cc:458] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d [sys.catalog]: This master's current role is: LEADER
I20260923 14:50:07.537703 21209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede4e20c4faa48d78cd7b53fefcbcd8d" member_type: VOTER last_known_addr { host: "127.30.2.254" port: 33893 } } peers { permanent_uuid: "636bf2fce43648ad871cf4cb37b7a6d0" member_type: VOTER last_known_addr { host: "127.30.2.253" port: 38757 } } peers { permanent_uuid: "e748f750acc8469d8dd19d3245612831" member_type: VOTER last_known_addr { host: "127.30.2.252" port: 32935 } } }
I20260923 14:50:07.537827 21209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:50:07.543895 21249 mvcc.cc:205] Tried to move back new op lower bound from 7332556830744944640 to 7332556830632493056. Current Snapshot: MvccSnapshot[applied={T|T < 7332556830744944640}]
I20260923 14:50:07.557588 21231 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:50:07.559521 21231 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:50:07.573971 21231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ede4e20c4faa48d78cd7b53fefcbcd8d: Generated new TSK 0
I20260923 14:50:07.574951 21231 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:50:07.617311 21192 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.612761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.30.2.252:32935)
0923 14:50:07.612881 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.612887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.614541 (+ 1654us) server_negotiation.cc:407] Connection header received
0923 14:50:07.615281 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.615285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.615349 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.615404 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.617159 (+ 1755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58630: BlockingRecv error: recv got EOF from 127.0.0.1:58630 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 14:50:07.617869 21193 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.612284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.30.2.254:33893)
0923 14:50:07.612431 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.612436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.614754 (+ 2318us) server_negotiation.cc:407] Connection header received
0923 14:50:07.615891 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.615894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.615951 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.616004 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.617762 (+ 1758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34624: BlockingRecv error: recv got EOF from 127.0.0.1:34624 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260923 14:50:07.618451 21114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.613281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.30.2.253:38757)
0923 14:50:07.613426 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.613431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.614957 (+ 1526us) server_negotiation.cc:407] Connection header received
0923 14:50:07.616471 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.616475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.616528 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.616579 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.618336 (+ 1757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35440: BlockingRecv error: recv got EOF from 127.0.0.1:35440 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 14:50:07.625825 21114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.624259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.30.2.253:38757)
0923 14:50:07.624386 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.624391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.624837 (+ 446us) server_negotiation.cc:407] Connection header received
0923 14:50:07.625066 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.625069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.625143 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.625196 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.625719 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35450: BlockingRecv error: recv got EOF from 127.0.0.1:35450 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 14:50:07.630980 21114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.629441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.30.2.253:38757)
0923 14:50:07.629533 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.629537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.630065 (+ 528us) server_negotiation.cc:407] Connection header received
0923 14:50:07.630262 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.630265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.630326 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.630377 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.630870 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35454: BlockingRecv error: recv got EOF from 127.0.0.1:35454 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260923 14:50:07.636526 21114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.634970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.30.2.253:38757)
0923 14:50:07.635065 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.635068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.635579 (+ 511us) server_negotiation.cc:407] Connection header received
0923 14:50:07.635762 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.635765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.635855 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.635907 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.636419 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35464: BlockingRecv error: recv got EOF from 127.0.0.1:35464 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260923 14:50:07.644227 21114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.642634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.30.2.253:38757)
0923 14:50:07.642978 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.642983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.643215 (+ 232us) server_negotiation.cc:407] Connection header received
0923 14:50:07.643433 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.643437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.643487 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.643543 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.644107 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35466: BlockingRecv error: recv got EOF from 127.0.0.1:35466 (error 108)
Metrics: {"server-negotiator.queue_time_us":264}
W20260923 14:50:07.656715 21114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.654793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.30.2.253:38757)
0923 14:50:07.654923 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.654928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.655583 (+ 655us) server_negotiation.cc:407] Connection header received
0923 14:50:07.655864 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.655868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.655931 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.655991 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.656572 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35470: BlockingRecv error: recv got EOF from 127.0.0.1:35470 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260923 14:50:07.677376 21114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.675311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.30.2.253:38757)
0923 14:50:07.675473 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.675478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.676206 (+ 728us) server_negotiation.cc:407] Connection header received
0923 14:50:07.676435 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.676439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.676510 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.676585 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.677239 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35476: BlockingRecv error: recv got EOF from 127.0.0.1:35476 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 14:50:07.713286 21114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.711967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.30.2.253:38757)
0923 14:50:07.712153 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.712164 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.712335 (+ 171us) server_negotiation.cc:407] Connection header received
0923 14:50:07.712509 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.712512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.712563 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.712635 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.713157 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35488: BlockingRecv error: recv got EOF from 127.0.0.1:35488 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 14:50:07.780850 21114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.779539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.30.2.253:38757)
0923 14:50:07.779724 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.779730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.779885 (+ 155us) server_negotiation.cc:407] Connection header received
0923 14:50:07.780070 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.780073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.780130 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.780223 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.780741 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35502: BlockingRecv error: recv got EOF from 127.0.0.1:35502 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 14:50:07.912637 21114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:07.911265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.30.2.253:38757)
0923 14:50:07.911441 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:07.911446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:07.911587 (+ 141us) server_negotiation.cc:407] Connection header received
0923 14:50:07.911838 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:07.911842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:07.911904 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:07.912005 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:07.912505 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35508: BlockingRecv error: recv got EOF from 127.0.0.1:35508 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 14:50:08.172397 21114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:50:08.170972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.30.2.253:38757)
0923 14:50:08.171134 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:08.171140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:08.171307 (+ 167us) server_negotiation.cc:407] Connection header received
0923 14:50:08.171504 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:08.171507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:08.171560 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:08.171651 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:50:08.172294 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35512: BlockingRecv error: recv got EOF from 127.0.0.1:35512 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:50:08.470620 21220 catalog_manager.cc:1259] Loaded cluster ID: 28dfa0f5983344ffbf0d51715e2596f1
I20260923 14:50:08.470748 21220 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0: loading cluster ID for follower catalog manager: success
I20260923 14:50:08.472491 21220 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0: acquiring CA information for follower catalog manager: success
I20260923 14:50:08.473701 21220 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 636bf2fce43648ad871cf4cb37b7a6d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:50:08.519920 21244 catalog_manager.cc:1259] Loaded cluster ID: 28dfa0f5983344ffbf0d51715e2596f1
I20260923 14:50:08.520001 21244 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831: loading cluster ID for follower catalog manager: success
I20260923 14:50:08.521827 21244 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831: acquiring CA information for follower catalog manager: success
I20260923 14:50:08.522996 21244 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e748f750acc8469d8dd19d3245612831: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:50:08.690546 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:08.686472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35520 (local address 127.30.2.253:38757)
0923 14:50:08.686714 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:08.686717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:08.686780 (+ 63us) server_negotiation.cc:407] Connection header received
0923 14:50:08.686950 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:08.686953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:08.687088 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:08.687216 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:50:08.687432 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:08.687994 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:08.688893 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:08.689155 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:08.689261 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:08.689588 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:50:08.690057 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:50:08.690156 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:08.690285 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 14:50:08.690339 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260923 14:50:08.697587 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:08.693738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.30.2.252:32935)
0923 14:50:08.693993 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:08.693998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:08.694014 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:50:08.694112 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:08.694116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:08.694251 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:08.694377 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:50:08.694598 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:08.695120 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:08.696021 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:08.696261 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:08.696376 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:08.696701 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:50:08.697159 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:50:08.697300 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:08.697362 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 14:50:08.697422 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260923 14:50:08.705468 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:08.700995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.30.2.253:38757)
0923 14:50:08.701247 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:08.701251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:08.701610 (+ 359us) server_negotiation.cc:407] Connection header received
0923 14:50:08.701760 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:08.701765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:08.701820 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:08.701887 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:50:08.702333 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:08.702848 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:08.704282 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:08.704589 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:08.704695 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:08.704939 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:50:08.705005 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:50:08.705077 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:08.705243 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 14:50:08.705305 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260923 14:50:08.705603 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:08.701071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.30.2.252:32935)
0923 14:50:08.701181 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:08.701213 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:08.701449 (+ 236us) server_negotiation.cc:407] Connection header received
0923 14:50:08.701664 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:08.701666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:08.701737 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:08.701803 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:50:08.702231 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:08.703063 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:08.704307 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:08.704670 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:08.704756 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:08.705099 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:50:08.705172 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:50:08.705261 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:08.705425 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 14:50:08.705478 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:50:08.705778 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:50:08.701221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.30.2.254:33893)
0923 14:50:08.701461 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 14:50:08.701465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:50:08.701480 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:50:08.701636 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:50:08.701639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:50:08.701803 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:50:08.701926 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:50:08.702323 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:08.703122 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:08.704018 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:50:08.704286 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:50:08.704383 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:50:08.704694 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:50:08.705042 (+ 348us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:50:08.705124 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 14:50:08.705282 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 14:50:08.705342 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
I20260923 14:50:08.708953 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 20965
I20260923 14:50:08.716331 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 21041
I20260923 14:50:08.722528 30731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknhhgz8/build/debug/bin/kudu with pid 21116
2026-09-23T14:50:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3124 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6261 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285374 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST